2024-01-16T16:07:24.520 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T16:07:24.520 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255 --verbose -- /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255/orig.config.yaml 2024-01-16T16:07:24.572 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T16:07:24.655 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255 branch: wip-yuri5-testing-2024-01-11-1300-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7518255' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-01-11-1300-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2024-01-11-1300-pacific sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5979 sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2024-01-11-1300-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f targets: smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCI5KUKyPq1BeVf1ZpBGEQk2yziu3e/9ybt2Y1uGQayLO2NHgYzXWz2umcQazanfvGyXCGDGmnehKnWACr4g9DQ= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZhfNIRpxTyWntFUjOkAB/4xt0JjrYVa5smZzrD8Dl+c71lylpOUdl7IoEp33XwzaS5Cw/24e5iW0TRVSSuwLs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-16_15:55:17 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-16T16:07:24.655 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa; will attempt to use it 2024-01-16T16:07:24.656 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks 2024-01-16T16:07:24.656 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T16:07:24.658 INFO:teuthology.task.internal:Checking packages... 2024-01-16T16:07:24.681 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a70aebef70df6ec2d957347ccbfc484e0573dd5f' 2024-01-16T16:07:24.681 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T16:07:24.681 INFO:teuthology.packaging:ref: None 2024-01-16T16:07:24.681 INFO:teuthology.packaging:tag: None 2024-01-16T16:07:24.682 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-01-11-1300-pacific 2024-01-16T16:07:24.682 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T16:07:24.682 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri5-testing-2024-01-11-1300-pacific 2024-01-16T16:07:24.830 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-449.ga70aebef 2024-01-16T16:07:24.832 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T16:07:24.838 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T16:07:24.838 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T16:07:24.852 INFO:teuthology.task.internal:Saving configuration 2024-01-16T16:07:24.871 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T16:07:24.877 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T16:07:24.902 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 15:58:32.976180', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCI5KUKyPq1BeVf1ZpBGEQk2yziu3e/9ybt2Y1uGQayLO2NHgYzXWz2umcQazanfvGyXCGDGmnehKnWACr4g9DQ='} 2024-01-16T16:07:24.923 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 15:58:32.977597', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZhfNIRpxTyWntFUjOkAB/4xt0JjrYVa5smZzrD8Dl+c71lylpOUdl7IoEp33XwzaS5Cw/24e5iW0TRVSSuwLs='} 2024-01-16T16:07:24.923 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T16:07:24.929 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-16T16:07:24.929 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-16T16:07:24.929 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T16:07:25.004 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f94ce023160>, signals=[15]) 2024-01-16T16:07:25.005 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T16:07:25.011 INFO:teuthology.task.internal:Opening connections... 2024-01-16T16:07:25.011 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-01-16T16:07:25.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T16:07:25.105 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-01-16T16:07:25.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T16:07:25.180 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T16:07:25.185 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-01-16T16:07:25.207 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-01-16T16:07:25.208 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-01-16T16:07:25.269 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-01-16T16:07:25.269 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="8" 2024-01-16T16:07:25.269 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-01-16T16:07:25.269 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-01-16T16:07:25.269 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="8" 2024-01-16T16:07:25.269 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el8" 2024-01-16T16:07:25.270 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-16T16:07:25.270 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-01-16T16:07:25.270 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-16T16:07:25.270 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-01-16T16:07:25.270 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T16:07:25.270 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-16T16:07:25.270 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T16:07:25.271 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-01-16T16:07:25.294 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-01-16T16:07:25.316 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-01-16T16:07:25.317 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="8" 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="8" 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el8" 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-01-16T16:07:25.378 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-16T16:07:25.379 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-01-16T16:07:25.379 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T16:07:25.379 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-16T16:07:25.379 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T16:07:25.379 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-01-16T16:07:25.402 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T16:07:25.411 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T16:07:25.416 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T16:07:25.416 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-01-16T16:07:25.419 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-01-16T16:07:25.438 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T16:07:25.444 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T16:07:25.444 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-01-16T16:07:25.479 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-01-16T16:07:25.506 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T16:07:25.564 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T16:07:25.579 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-16T16:07:25.580 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-16T16:07:25.580 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T16:07:25.580 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T16:07:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T16:07:25.600 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-01-16T16:07:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T16:07:25.601 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-01-16T16:07:25.659 INFO:teuthology.orchestra.run.smithi154.stdout:4.18.0-489.el8.x86_64 2024-01-16T16:07:25.659 INFO:teuthology.task.kernel:Running kernel on smithi154: 4.18.0-489.el8.x86_64 2024-01-16T16:07:25.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2024-01-16T16:07:25.660 INFO:teuthology.orchestra.run.smithi190.stdout:4.18.0-489.el8.x86_64 2024-01-16T16:07:25.661 INFO:teuthology.task.kernel:Running kernel on smithi190: 4.18.0-489.el8.x86_64 2024-01-16T16:07:25.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-01-16T16:53:42.625 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - AppStream 13 kB/s | 35 MB 46:14 2024-01-16T16:53:51.391 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - BaseOS 26 MB/s | 59 MB 00:02 2024-01-16T16:54:00.440 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - Extras 30 kB/s | 18 kB 00:00 2024-01-16T16:54:01.215 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2024-01-16T16:54:01.667 INFO:teuthology.orchestra.run.smithi190.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-01-16T16:54:45.164 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 381 kB/s | 16 MB 00:43 2024-01-16T16:54:49.037 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 381 kB/s | 24 kB 00:00 2024-01-16T16:54:52.833 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-16T16:54:52.833 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-16T16:54:53.131 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T16:54:53.132 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T16:54:53.132 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T16:54:53.132 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T16:54:53.132 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T16:54:53.132 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-16T16:54:53.132 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T16:54:53.133 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-16T16:54:53.133 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-16T16:54:53.133 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T16:54:53.133 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T16:54:53.133 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T16:54:53.133 INFO:teuthology.orchestra.run.smithi190.stdout:Install 3 Packages 2024-01-16T16:54:53.133 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T16:54:53.134 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 89 M 2024-01-16T16:54:53.134 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 96 M 2024-01-16T16:54:53.134 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T16:54:54.421 INFO:teuthology.orchestra.run.smithi190.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 8.7 MB/s | 10 MB 00:01 2024-01-16T16:54:55.596 INFO:teuthology.orchestra.run.smithi190.stdout:(2/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-01-16T16:54:56.222 INFO:teuthology.orchestra.run.smithi190.stdout:(3/3): kernel-core-4.18.0-535.el8.x86_64.rpm 14 MB/s | 43 MB 00:02 2024-01-16T16:54:56.222 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T16:54:56.222 INFO:teuthology.orchestra.run.smithi190.stdout:Total 29 MB/s | 89 MB 00:03 2024-01-16T16:54:56.899 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T16:54:57.324 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T16:54:57.324 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T16:54:58.577 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T16:54:58.607 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T16:55:01.369 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T16:55:01.811 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-16T16:55:05.383 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-16T16:55:05.776 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-16T16:55:09.198 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-16T16:55:09.465 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:55:31.305 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:55:31.329 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:55:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:55:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-16T16:55:31.678 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-16T16:55:32.488 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:55:32.488 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T16:55:32.488 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T16:55:32.488 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-16T16:55:32.488 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-16T16:55:32.488 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T16:55:32.489 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T16:55:32.886 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T16:55:33.104 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-16T16:55:33.105 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-16T16:55:33.105 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-16T16:55:33.105 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-16T16:55:33.105 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-16T16:55:33.105 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-16T16:55:33.105 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-01-16T16:55:33.105 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T16:55:33.315 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-16T16:55:33.315 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-16T16:55:33.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-01-16T16:55:33.349 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.02 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 129.el8 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Group : System Environment/Base 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 9265586 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-16T16:55:33.350 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:Relocations : (not relocatable) 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : CentOS BuildSystem 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T16:55:33.351 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-01-16T16:55:33.352 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T16:55:33.352 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T16:55:33.353 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T16:55:33.353 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 4.18.0-535.el8.x86_64 2024-01-16T16:55:33.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T16:55:34.748 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-01-16T16:55:36.927 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-01-16T16:55:36.929 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-01-16T16:55:36.951 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.TjkTY8Smwo 2024-01-16T16:55:36.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TjkTY8Smwo 2024-01-16T16:55:37.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.TjkTY8Smwo 2024-01-16T16:55:37.180 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.TjkTY8Smwo is 6KB 2024-01-16T16:55:37.228 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.TjkTY8Smwo 2024-01-16T16:55:37.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T16:55:37.311 INFO:teuthology.orchestra.run.smithi190.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-16T16:55:37.311 INFO:teuthology.orchestra.run.smithi190.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-16T16:55:37.311 INFO:teuthology.orchestra.run.smithi190.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-16T16:55:37.311 INFO:teuthology.orchestra.run.smithi190.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-16T16:55:37.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-16T16:55:37.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-01-16T16:56:04.898 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - AppStream 12 kB/s | 35 MB 48:36 2024-01-16T16:56:07.497 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-16T16:56:07.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-01-16T16:56:07.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T16:56:14.089 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 59 MB 00:02 2024-01-16T16:56:23.296 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - Extras 36 kB/s | 18 kB 00:00 2024-01-16T16:56:24.029 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2024-01-16T16:56:24.524 INFO:teuthology.orchestra.run.smithi154.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2024-01-16T16:56:27.735 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 16 MB 00:03 2024-01-16T16:56:31.598 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 402 kB/s | 24 kB 00:00 2024-01-16T16:56:35.382 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-16T16:56:35.382 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-16T16:56:35.689 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T16:56:35.689 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 4.18.0-535.el8 baseos 10 M 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 4.18.0-535.el8 baseos 43 M 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 4.18.0-535.el8 baseos 36 M 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout:Install 3 Packages 2024-01-16T16:56:35.690 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T16:56:35.691 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 89 M 2024-01-16T16:56:35.691 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 96 M 2024-01-16T16:56:35.691 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T16:56:37.291 INFO:teuthology.orchestra.run.smithi154.stdout:(1/3): kernel-4.18.0-535.el8.x86_64.rpm 6.8 MB/s | 10 MB 00:01 2024-01-16T16:56:38.791 INFO:teuthology.orchestra.run.smithi154.stdout:(2/3): kernel-core-4.18.0-535.el8.x86_64.rpm 14 MB/s | 43 MB 00:02 2024-01-16T16:56:40.424 INFO:teuthology.orchestra.run.smithi154.stdout:(3/3): kernel-modules-4.18.0-535.el8.x86_64.rpm 7.8 MB/s | 36 MB 00:04 2024-01-16T16:56:40.425 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T16:56:40.425 INFO:teuthology.orchestra.run.smithi154.stdout:Total 19 MB/s | 89 MB 00:04 2024-01-16T16:56:41.101 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T16:56:41.535 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T16:56:41.535 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T16:56:42.044 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-01-16T16:56:42.777 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T16:56:42.809 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T16:56:45.614 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T16:56:46.055 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-16T16:56:49.690 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 1/3 2024-01-16T16:56:50.071 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-16T16:56:51.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-01-16T16:56:51.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T16:56:53.512 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 2/3 2024-01-16T16:56:53.784 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:57:09.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-01-16T16:57:15.715 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:57:15.738 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:57:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:57:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-4.18.0-535.el8.x86_64 1/3 2024-01-16T16:57:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-4.18.0-535.el8.x86_64 2/3 2024-01-16T16:57:16.871 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-4.18.0-535.el8.x86_64 3/3 2024-01-16T16:57:16.871 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T16:57:16.871 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T16:57:16.871 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-4.18.0-535.el8.x86_64 kernel-core-4.18.0-535.el8.x86_64 2024-01-16T16:57:16.871 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-4.18.0-535.el8.x86_64 2024-01-16T16:57:16.871 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T16:57:16.871 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T16:57:17.550 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T16:57:17.773 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-16T16:57:17.774 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-16T16:57:17.774 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-535.el8.x86_64 2024-01-16T16:57:17.774 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-16T16:57:17.774 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-16T16:57:17.775 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-16T16:57:17.775 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-01-16T16:57:17.775 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-01-16T16:57:17.986 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-4.18.0-535.el8.x86_64 2024-01-16T16:57:17.986 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-535.el8.x86_64 2024-01-16T16:57:17.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2024-01-16T16:57:18.019 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.02 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 129.el8 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Group : System Environment/Base 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 9265586 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-16T16:57:18.020 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout:Relocations : (not relocatable) 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : CentOS BuildSystem 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T16:57:18.021 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T16:57:18.022 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T16:57:18.022 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2024-01-16T16:57:18.022 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T16:57:18.022 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T16:57:18.023 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T16:57:18.023 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 4.18.0-535.el8.x86_64 2024-01-16T16:57:18.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T16:57:19.320 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-01-16T16:57:21.418 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-01-16T16:57:21.419 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-01-16T16:57:21.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-01-16T16:57:21.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T16:57:21.451 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.54ItsYQjR3 2024-01-16T16:57:21.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.54ItsYQjR3 2024-01-16T16:57:21.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.54ItsYQjR3 2024-01-16T16:57:21.689 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.54ItsYQjR3 is 6KB 2024-01-16T16:57:21.736 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.54ItsYQjR3 2024-01-16T16:57:21.746 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-01-16T16:57:21.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T16:57:21.825 INFO:teuthology.orchestra.run.smithi154.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-16T16:57:21.825 INFO:teuthology.orchestra.run.smithi154.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-16T16:57:21.825 INFO:teuthology.orchestra.run.smithi154.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-16T16:57:21.825 INFO:teuthology.orchestra.run.smithi154.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64.conf 2024-01-16T16:57:21.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-535.el8.x86_64 2024-01-16T16:57:22.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-01-16T16:57:22.048 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-01-16T16:57:22.048 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-535.el8.x86_64"... 2024-01-16T16:57:22.048 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-01-16T16:57:22.065 INFO:teuthology.orchestra.run.smithi190.stdout:4.18.0-535.el8.x86_64 2024-01-16T16:57:22.065 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-16T16:57:22.065 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T16:57:22.065 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T16:57:23.066 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-16T16:57:23.066 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T16:57:23.182 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-01-16T16:57:23.197 DEBUG:teuthology.parallel:result is None 2024-01-16T16:57:52.012 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-16T16:57:52.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-01-16T16:57:52.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T16:58:26.492 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-01-16T16:58:35.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-01-16T16:58:35.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T16:59:35.496 DEBUG:teuthology.orchestra.remote:timed out 2024-01-16T16:59:47.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-01-16T16:59:47.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T16:59:47.837 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-01-16T16:59:48.139 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-01-16T16:59:48.139 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-535.el8.x86_64"... 2024-01-16T16:59:48.139 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-01-16T16:59:48.158 INFO:teuthology.orchestra.run.smithi154.stdout:4.18.0-535.el8.x86_64 2024-01-16T16:59:48.158 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-535.el8.x86_64 vs 4.18.0-535.el8.x86_64 2024-01-16T16:59:48.158 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T16:59:48.158 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T16:59:49.160 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-16T16:59:49.160 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T16:59:49.248 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2024-01-16T16:59:49.260 DEBUG:teuthology.parallel:result is None 2024-01-16T16:59:49.260 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T16:59:49.268 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T16:59:49.268 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T16:59:49.271 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T16:59:49.295 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T16:59:49.302 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T16:59:49.309 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T16:59:49.310 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T16:59:49.334 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T16:59:49.395 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T16:59:49.402 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T16:59:49.403 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T16:59:49.407 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T16:59:49.445 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T16:59:49.448 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T16:59:49.455 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T16:59:49.459 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T16:59:49.461 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T16:59:49.468 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T16:59:49.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T16:59:49.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T16:59:49.532 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T16:59:49.542 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T16:59:49.542 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T16:59:49.568 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T16:59:49.592 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T16:59:49.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T16:59:49.723 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T16:59:49.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T16:59:49.857 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T16:59:49.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T16:59:49.924 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T16:59:49.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T16:59:50.023 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T16:59:50.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T16:59:50.154 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-01-16T16:59:50.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T16:59:50.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-01-16T16:59:50.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-01-16T16:59:50.282 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T16:59:50.326 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T16:59:50.728 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T16:59:50.736 INFO:teuthology.task.internal:Starting timer... 2024-01-16T16:59:50.736 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T16:59:50.747 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T16:59:50.756 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-16T16:59:50.756 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-01-16T16:59:50.843 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-01-16T16:59:50.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-01-16T16:59:50.927 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-01-16T16:59:50.929 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T16:59:50.929 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-01-16T16:59:50.969 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-01-16T16:59:50.969 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-01-16T16:59:51.013 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-01-16T16:59:51.013 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi154.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-01-16T16:59:51.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T16:59:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T16:59:51.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T16:59:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T16:59:51.081 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T16:59:51.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-01-16T16:59:51.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-01-16T16:59:51.149 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T16:59:51.159 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T16:59:51.176 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T16:59:51.178 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi154.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T17:05:38.472 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi154.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-01-16T17:05:38.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-01-16T17:05:38.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T17:05:38.546 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-01-16T17:05:38.613 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-01-16T17:05:38.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-01-16T17:05:38.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T17:05:38.688 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-01-16T17:05:38.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-01-16T17:05:38.762 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T17:05:38.773 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T17:05:38.773 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T17:05:38.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T17:05:38.776 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T17:05:38.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T17:05:38.805 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T17:05:38.819 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T17:05:38.833 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T17:05:38.847 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T17:05:38.897 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-01-16T17:05:38.908 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-01-16T17:05:38.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-01-16T17:05:38.921 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T17:05:38.925 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-01-16T17:05:38.934 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T17:05:38.939 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T17:05:38.952 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T17:05:38.997 INFO:teuthology.orchestra.run.smithi154.stderr:bash: ntpq: command not found 2024-01-16T17:05:39.001 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T17:05:39.001 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-01-16T17:05:39.001 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:05:39.001 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:05:39.001 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:05:39.001 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:05:39.026 INFO:teuthology.orchestra.run.smithi190.stderr:bash: ntpq: command not found 2024-01-16T17:05:39.029 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T17:05:39.029 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-01-16T17:05:39.029 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:05:39.029 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:05:39.029 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:05:39.029 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:05:39.030 INFO:teuthology.run_tasks:Running task pexec... 2024-01-16T17:05:39.039 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-16T17:05:39.040 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi154.front.sepia.ceph.com 2024-01-16T17:05:39.040 DEBUG:teuthology.orchestra.run.smithi154:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-16T17:05:39.040 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi190.front.sepia.ceph.com 2024-01-16T17:05:39.040 DEBUG:teuthology.orchestra.run.smithi190:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-16T17:05:39.923 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:25 ago on Tue 16 Jan 2024 05:04:14 PM UTC. 2024-01-16T17:05:39.954 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:25 ago on Tue 16 Jan 2024 05:04:14 PM UTC. 2024-01-16T17:05:42.180 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout:Resetting modules: 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:05:42.181 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:05:42.189 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:05:42.190 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:05:42.190 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:05:42.190 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:05:42.190 INFO:teuthology.orchestra.run.smithi154.stdout:Resetting modules: 2024-01-16T17:05:42.190 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools 2024-01-16T17:05:42.190 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:05:42.190 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:05:42.190 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:05:42.191 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:05:42.434 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:05:42.441 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:05:43.376 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:29 ago on Tue 16 Jan 2024 05:04:14 PM UTC. 2024-01-16T17:05:43.391 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:29 ago on Tue 16 Jan 2024 05:04:14 PM UTC. 2024-01-16T17:05:45.559 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:05:45.623 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:05:45.760 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:05:45.760 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-01-16T17:05:45.760 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:05:45.760 INFO:teuthology.orchestra.run.smithi190.stdout:Installing group/module packages: 2024-01-16T17:05:45.760 INFO:teuthology.orchestra.run.smithi190.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-16T17:05:45.760 INFO:teuthology.orchestra.run.smithi190.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-16T17:05:45.760 INFO:teuthology.orchestra.run.smithi190.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-16T17:05:45.760 INFO:teuthology.orchestra.run.smithi190.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout:Installing module profiles: 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools/common 2024-01-16T17:05:45.761 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:05:45.762 INFO:teuthology.orchestra.run.smithi190.stdout:Enabling module streams: 2024-01-16T17:05:45.762 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools rhel8 2024-01-16T17:05:45.762 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:05:45.762 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:05:45.762 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:05:45.762 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-01-16T17:05:45.762 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:05:45.763 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 27 M 2024-01-16T17:05:45.763 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 89 M 2024-01-16T17:05:45.763 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:05:45.828 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:05:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-01-16T17:05:45.828 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:05:45.828 INFO:teuthology.orchestra.run.smithi154.stdout:Installing group/module packages: 2024-01-16T17:05:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-16T17:05:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-16T17:05:45.829 INFO:teuthology.orchestra.run.smithi154.stdout:Installing module profiles: 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools/common 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout:Enabling module streams: 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: container-tools rhel8 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2024-01-16T17:05:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:05:45.831 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 27 M 2024-01-16T17:05:45.831 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 89 M 2024-01-16T17:05:45.831 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:05:46.298 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.1 MB/s | 750 kB 00:00 2024-01-16T17:05:46.340 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.6 MB/s | 1.0 MB 00:00 2024-01-16T17:05:46.396 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.9 MB/s | 750 kB 00:00 2024-01-16T17:05:46.399 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 377 kB/s | 38 kB 00:00 2024-01-16T17:05:46.429 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.4 MB/s | 1.0 MB 00:00 2024-01-16T17:05:46.441 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.3 MB/s | 241 kB 00:00 2024-01-16T17:05:46.471 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 502 kB/s | 38 kB 00:00 2024-01-16T17:05:46.541 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2024-01-16T17:05:46.555 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 1.9 MB/s | 241 kB 00:00 2024-01-16T17:05:46.617 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-01-16T17:05:46.655 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.7 MB/s | 178 kB 00:00 2024-01-16T17:05:46.725 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 866 kB/s | 94 kB 00:00 2024-01-16T17:05:46.739 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 297 kB/s | 25 kB 00:00 2024-01-16T17:05:46.831 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2024-01-16T17:05:46.859 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 11 MB/s | 9.6 MB 00:00 2024-01-16T17:05:46.934 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 7.3 MB/s | 3.9 MB 00:00 2024-01-16T17:05:46.982 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 10 MB/s | 9.6 MB 00:00 2024-01-16T17:05:47.001 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 18 MB/s | 2.5 MB 00:00 2024-01-16T17:05:47.018 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 630 kB/s | 53 kB 00:00 2024-01-16T17:05:47.040 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 6.9 MB/s | 3.9 MB 00:00 2024-01-16T17:05:47.077 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 544 kB/s | 41 kB 00:00 2024-01-16T17:05:47.132 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 17 MB/s | 2.5 MB 00:00 2024-01-16T17:05:47.149 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 484 kB/s | 53 kB 00:00 2024-01-16T17:05:47.199 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 613 kB/s | 41 kB 00:00 2024-01-16T17:05:47.302 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 14 MB/s | 8.1 MB 00:00 2024-01-16T17:05:47.302 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:05:47.303 INFO:teuthology.orchestra.run.smithi190.stdout:Total 17 MB/s | 27 MB 00:01 2024-01-16T17:05:47.425 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 14 MB/s | 8.1 MB 00:00 2024-01-16T17:05:47.425 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:05:47.425 INFO:teuthology.orchestra.run.smithi154.stdout:Total 17 MB/s | 27 MB 00:01 2024-01-16T17:05:47.571 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:05:47.592 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:05:47.592 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:05:47.694 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:05:47.715 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:05:47.716 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:05:47.849 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:05:47.852 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:05:47.985 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:05:47.989 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:05:48.534 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:05:48.690 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:05:48.754 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-16T17:05:48.906 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-16T17:05:48.932 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-16T17:05:49.068 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-16T17:05:49.164 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-16T17:05:49.295 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-16T17:05:49.403 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-16T17:05:49.503 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-16T17:05:49.995 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-16T17:05:50.034 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-16T17:05:51.002 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-16T17:05:51.059 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-16T17:05:51.238 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-16T17:05:51.279 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-16T17:05:51.480 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-16T17:05:51.495 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-16T17:05:51.728 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-16T17:05:51.747 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-16T17:05:52.150 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-16T17:05:52.170 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-16T17:05:53.035 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-16T17:05:53.079 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-16T17:05:53.241 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-16T17:05:53.304 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-16T17:05:53.417 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-16T17:05:53.490 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-16T17:05:53.842 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-16T17:05:53.843 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-16T17:05:53.914 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-16T17:05:53.914 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-16T17:05:53.914 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-16T17:05:53.914 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-16T17:05:53.914 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-16T17:05:53.914 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-16T17:05:53.914 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-16T17:05:53.915 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-16T17:05:53.915 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-16T17:05:53.915 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-16T17:05:53.915 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-16T17:05:53.915 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-16T17:05:53.915 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-16T17:05:54.668 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-16T17:05:54.669 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-16T17:05:54.669 INFO:teuthology.orchestra.run.smithi154.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.669 INFO:teuthology.orchestra.run.smithi154.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.669 INFO:teuthology.orchestra.run.smithi154.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-16T17:05:54.669 INFO:teuthology.orchestra.run.smithi154.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-16T17:05:54.669 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:05:54.669 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:05:54.766 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-16T17:05:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-16T17:05:54.768 INFO:teuthology.orchestra.run.smithi190.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-16T17:05:54.768 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:05:54.768 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:05:54.916 DEBUG:teuthology.parallel:result is None 2024-01-16T17:05:55.054 DEBUG:teuthology.parallel:result is None 2024-01-16T17:05:55.054 INFO:teuthology.run_tasks:Running task pexec... 2024-01-16T17:05:55.062 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-16T17:05:55.063 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi154.front.sepia.ceph.com 2024-01-16T17:05:55.063 DEBUG:teuthology.orchestra.run.smithi154:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-16T17:05:55.063 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi190.front.sepia.ceph.com 2024-01-16T17:05:55.064 DEBUG:teuthology.orchestra.run.smithi190:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-16T17:05:55.272 DEBUG:teuthology.parallel:result is None 2024-01-16T17:05:55.286 DEBUG:teuthology.parallel:result is None 2024-01-16T17:05:55.286 INFO:teuthology.run_tasks:Running task install... 2024-01-16T17:05:55.294 DEBUG:teuthology.task.install:project ceph 2024-01-16T17:05:55.294 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'}} 2024-01-16T17:05:55.294 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'} 2024-01-16T17:05:55.294 INFO:teuthology.task.install:Using flavor: default 2024-01-16T17:05:55.302 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T17:05:55.302 INFO:teuthology.task.install:extra packages: [] 2024-01-16T17:05:55.302 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-16T17:05:55.302 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-16T17:05:55.302 INFO:teuthology.packaging:ref: None 2024-01-16T17:05:55.303 INFO:teuthology.packaging:tag: v16.2.4 2024-01-16T17:05:55.303 INFO:teuthology.packaging:branch: None 2024-01-16T17:05:55.303 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T17:05:55.569 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-16T17:05:55.569 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-16T17:05:55.571 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-16T17:05:55.571 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-16T17:05:55.572 INFO:teuthology.packaging:ref: None 2024-01-16T17:05:55.572 INFO:teuthology.packaging:tag: v16.2.4 2024-01-16T17:05:55.572 INFO:teuthology.packaging:branch: None 2024-01-16T17:05:55.572 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T17:05:55.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-16T17:05:55.712 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-16T17:05:55.713 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-16T17:05:55.739 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T17:05:55.739 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:05:55.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T17:05:55.768 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-16T17:05:55.768 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-16T17:05:55.768 INFO:teuthology.packaging:ref: None 2024-01-16T17:05:55.768 INFO:teuthology.packaging:tag: v16.2.4 2024-01-16T17:05:55.768 INFO:teuthology.packaging:branch: None 2024-01-16T17:05:55.768 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T17:05:55.768 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T17:05:55.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T17:05:55.878 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-16T17:05:55.878 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-16T17:05:55.901 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T17:05:55.901 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-01-16T17:05:55.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T17:05:55.912 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T17:05:55.930 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-16T17:05:55.930 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-16T17:05:55.931 INFO:teuthology.packaging:ref: None 2024-01-16T17:05:55.931 INFO:teuthology.packaging:tag: v16.2.4 2024-01-16T17:05:55.931 INFO:teuthology.packaging:branch: None 2024-01-16T17:05:55.931 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T17:05:55.931 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T17:05:55.979 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-01-16T17:05:55.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-01-16T17:05:55.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T17:05:56.076 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T17:05:56.143 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-01-16T17:05:56.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-01-16T17:05:56.253 INFO:teuthology.orchestra.run.smithi154.stdout:76 files removed 2024-01-16T17:05:56.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw 2024-01-16T17:05:56.424 INFO:teuthology.orchestra.run.smithi190.stdout:76 files removed 2024-01-16T17:05:56.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-01-16T17:05:56.775 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 915 kB/s | 74 kB 00:00 2024-01-16T17:05:56.884 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 213 kB/s | 15 kB 00:00 2024-01-16T17:05:56.935 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 982 kB/s | 74 kB 00:00 2024-01-16T17:05:57.002 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2024-01-16T17:05:57.110 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 111 kB/s | 15 kB 00:00 2024-01-16T17:05:57.211 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-01-16T17:05:58.895 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - AppStream 19 MB/s | 35 MB 00:01 2024-01-16T17:05:59.237 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - AppStream 18 MB/s | 35 MB 00:01 2024-01-16T17:06:15.344 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - BaseOS 6.0 MB/s | 59 MB 00:09 2024-01-16T17:06:16.311 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - BaseOS 5.6 MB/s | 59 MB 00:10 2024-01-16T17:06:24.644 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - Extras 23 kB/s | 18 kB 00:00 2024-01-16T17:06:25.228 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2024-01-16T17:06:25.495 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - Extras 28 kB/s | 18 kB 00:00 2024-01-16T17:06:26.087 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2024-01-16T17:06:27.679 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 8 - PowerTools 2.9 MB/s | 6.8 MB 00:02 2024-01-16T17:06:28.746 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 8 - PowerTools 2.7 MB/s | 6.8 MB 00:02 2024-01-16T17:06:29.297 INFO:teuthology.orchestra.run.smithi154.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2024-01-16T17:06:30.197 INFO:teuthology.orchestra.run.smithi190.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-16T17:06:30.289 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-01-16T17:06:32.526 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 16 MB 00:02 2024-01-16T17:06:34.164 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 468 kB/s | 24 kB 00:00 2024-01-16T17:06:36.515 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 410 kB/s | 24 kB 00:00 2024-01-16T17:06:38.900 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:06:38.903 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:06:38.903 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-16T17:06:38.904 INFO:teuthology.orchestra.run.smithi154.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-16T17:06:38.905 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout:Install 25 Packages 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 2 Packages 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:06:38.906 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 59 M 2024-01-16T17:06:38.907 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:06:39.641 INFO:teuthology.orchestra.run.smithi154.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 35 MB/s | 6.0 MB 00:00 2024-01-16T17:06:39.666 INFO:teuthology.orchestra.run.smithi154.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 999 kB/s | 25 kB 00:00 2024-01-16T17:06:39.708 INFO:teuthology.orchestra.run.smithi154.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-16T17:06:39.742 INFO:teuthology.orchestra.run.smithi154.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-16T17:06:39.900 INFO:teuthology.orchestra.run.smithi154.stdout:(5/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 29 MB/s | 13 MB 00:00 2024-01-16T17:06:39.917 INFO:teuthology.orchestra.run.smithi154.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-01-16T17:06:39.942 INFO:teuthology.orchestra.run.smithi154.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-16T17:06:40.010 INFO:teuthology.orchestra.run.smithi154.stdout:(8/27): librgw2-16.2.4-0.el8.x86_64.rpm 15 MB/s | 4.1 MB 00:00 2024-01-16T17:06:40.035 INFO:teuthology.orchestra.run.smithi154.stdout:(9/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 213 kB 00:00 2024-01-16T17:06:40.060 INFO:teuthology.orchestra.run.smithi154.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 7.5 MB/s | 387 kB 00:00 2024-01-16T17:06:40.085 INFO:teuthology.orchestra.run.smithi154.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-16T17:06:40.111 INFO:teuthology.orchestra.run.smithi154.stdout:(12/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 4.8 MB/s | 368 kB 00:00 2024-01-16T17:06:40.419 INFO:teuthology.orchestra.run.smithi154.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-01-16T17:06:40.445 INFO:teuthology.orchestra.run.smithi154.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 404 kB/s | 145 kB 00:00 2024-01-16T17:06:40.512 INFO:teuthology.orchestra.run.smithi154.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 658 kB/s | 44 kB 00:00 2024-01-16T17:06:40.595 INFO:teuthology.orchestra.run.smithi154.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 505 kB/s | 42 kB 00:00 2024-01-16T17:06:40.629 INFO:teuthology.orchestra.run.smithi154.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2024-01-16T17:06:40.654 INFO:teuthology.orchestra.run.smithi154.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 805 kB/s | 47 kB 00:00 2024-01-16T17:06:40.680 INFO:teuthology.orchestra.run.smithi154.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-01-16T17:06:40.738 INFO:teuthology.orchestra.run.smithi154.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 735 kB/s | 80 kB 00:00 2024-01-16T17:06:41.209 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:06:41.212 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:06:41.212 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-01-16T17:06:41.212 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-16T17:06:41.213 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-16T17:06:41.214 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout:Install 25 Packages 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-01-16T17:06:41.215 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:06:41.216 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 59 M 2024-01-16T17:06:41.216 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:06:41.297 INFO:teuthology.orchestra.run.smithi154.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 181 kB/s | 116 kB 00:00 2024-01-16T17:06:41.406 INFO:teuthology.orchestra.run.smithi154.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 271 kB/s | 181 kB 00:00 2024-01-16T17:06:41.431 INFO:teuthology.orchestra.run.smithi154.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 408 kB/s | 306 kB 00:00 2024-01-16T17:06:41.448 INFO:teuthology.orchestra.run.smithi154.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 391 kB/s | 59 kB 00:00 2024-01-16T17:06:41.507 INFO:teuthology.orchestra.run.smithi190.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 39 MB/s | 6.0 MB 00:00 2024-01-16T17:06:41.532 INFO:teuthology.orchestra.run.smithi154.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2024-01-16T17:06:41.533 INFO:teuthology.orchestra.run.smithi190.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-01-16T17:06:41.548 INFO:teuthology.orchestra.run.smithi154.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 528 kB/s | 75 kB 00:00 2024-01-16T17:06:41.575 INFO:teuthology.orchestra.run.smithi190.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-01-16T17:06:41.607 INFO:teuthology.orchestra.run.smithi154.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 26 MB/s | 4.0 MB 00:00 2024-01-16T17:06:41.608 INFO:teuthology.orchestra.run.smithi190.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-16T17:06:41.612 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:06:41.612 INFO:teuthology.orchestra.run.smithi154.stdout:Total 22 MB/s | 59 MB 00:02 2024-01-16T17:06:41.683 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:06:41.700 INFO:teuthology.orchestra.run.smithi190.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2024-01-16T17:06:41.717 INFO:teuthology.orchestra.run.smithi190.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-01-16T17:06:41.733 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:06:41.733 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:06:41.743 INFO:teuthology.orchestra.run.smithi190.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-16T17:06:41.768 INFO:teuthology.orchestra.run.smithi190.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-01-16T17:06:41.793 INFO:teuthology.orchestra.run.smithi190.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2024-01-16T17:06:41.835 INFO:teuthology.orchestra.run.smithi190.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 8.6 MB/s | 368 kB 00:00 2024-01-16T17:06:41.860 INFO:teuthology.orchestra.run.smithi190.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-16T17:06:42.036 INFO:teuthology.orchestra.run.smithi190.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-01-16T17:06:42.075 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:06:42.079 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:06:42.278 INFO:teuthology.orchestra.run.smithi190.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-01-16T17:06:42.303 INFO:teuthology.orchestra.run.smithi190.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 328 kB/s | 145 kB 00:00 2024-01-16T17:06:42.428 INFO:teuthology.orchestra.run.smithi190.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 351 kB/s | 44 kB 00:00 2024-01-16T17:06:42.612 INFO:teuthology.orchestra.run.smithi190.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 230 kB/s | 42 kB 00:00 2024-01-16T17:06:42.662 INFO:teuthology.orchestra.run.smithi190.stdout:(17/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 942 kB/s | 47 kB 00:00 2024-01-16T17:06:42.729 INFO:teuthology.orchestra.run.smithi190.stdout:(18/27): ndctl-libs-71.1-7.el8.x86_64.rpm 1.2 MB/s | 80 kB 00:00 2024-01-16T17:06:42.754 INFO:teuthology.orchestra.run.smithi190.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 545 kB/s | 259 kB 00:00 2024-01-16T17:06:42.780 INFO:teuthology.orchestra.run.smithi190.stdout:(20/27): librdkafka-1.6.1-1.el8.x86_64.rpm 910 kB/s | 677 kB 00:00 2024-01-16T17:06:42.905 INFO:teuthology.orchestra.run.smithi190.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 663 kB/s | 116 kB 00:00 2024-01-16T17:06:42.939 INFO:teuthology.orchestra.run.smithi190.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2024-01-16T17:06:42.964 INFO:teuthology.orchestra.run.smithi190.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 984 kB/s | 181 kB 00:00 2024-01-16T17:06:42.989 INFO:teuthology.orchestra.run.smithi190.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 702 kB/s | 59 kB 00:00 2024-01-16T17:06:43.015 INFO:teuthology.orchestra.run.smithi190.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 993 kB/s | 75 kB 00:00 2024-01-16T17:06:43.128 INFO:teuthology.orchestra.run.smithi190.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 3.7 MB 00:00 2024-01-16T17:06:43.198 INFO:teuthology.orchestra.run.smithi190.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 19 MB/s | 4.0 MB 00:00 2024-01-16T17:06:43.201 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:06:43.201 INFO:teuthology.orchestra.run.smithi190.stdout:Total 30 MB/s | 59 MB 00:01 2024-01-16T17:06:43.206 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:06:43.273 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:06:43.321 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:06:43.321 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:06:43.412 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-16T17:06:43.608 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-16T17:06:43.660 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:06:43.663 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:06:43.775 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-16T17:06:44.195 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-16T17:06:44.400 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-16T17:06:44.507 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-16T17:06:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-16T17:06:44.833 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:06:44.933 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-16T17:06:45.003 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-16T17:06:45.090 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-16T17:06:45.217 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-16T17:06:45.285 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-16T17:06:45.482 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-16T17:06:45.751 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-16T17:06:45.872 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-16T17:06:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-16T17:06:45.993 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-16T17:06:46.082 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-16T17:06:46.155 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-16T17:06:46.174 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-16T17:06:46.324 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-16T17:06:46.392 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-16T17:06:46.399 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-16T17:06:46.569 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-16T17:06:46.590 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-16T17:06:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-16T17:06:46.791 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-16T17:06:46.858 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-16T17:06:46.966 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-16T17:06:47.044 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-16T17:06:47.212 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-16T17:06:47.392 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-16T17:06:47.548 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-16T17:06:47.625 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-16T17:06:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-16T17:06:47.816 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-16T17:06:47.828 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-16T17:06:47.911 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-16T17:06:48.006 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-16T17:06:48.057 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-16T17:06:48.061 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-16T17:06:48.203 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-16T17:06:48.259 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-16T17:06:48.407 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-16T17:06:48.407 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-16T17:06:48.514 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-16T17:06:48.581 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-16T17:06:48.642 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-16T17:06:48.699 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-16T17:06:48.811 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-16T17:06:48.882 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-16T17:06:49.064 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-16T17:06:49.281 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-16T17:06:49.308 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-16T17:06:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-16T17:06:49.550 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-16T17:06:49.718 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-16T17:06:49.941 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-16T17:06:50.106 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-16T17:06:50.184 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-16T17:06:50.374 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-16T17:06:50.657 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-16T17:06:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-16T17:06:51.048 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-16T17:06:51.848 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-16T17:06:52.107 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-16T17:06:52.808 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-16T17:06:53.118 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-16T17:06:53.206 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-16T17:06:53.339 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-16T17:06:53.572 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-16T17:06:53.798 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-16T17:06:54.504 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-16T17:06:54.821 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-16T17:06:54.917 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-16T17:06:55.097 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-16T17:07:12.508 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /dev 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /run 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /dev 2024-01-16T17:07:12.509 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:13.881 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-16T17:07:13.943 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-16T17:07:13.943 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-16T17:07:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-16T17:07:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-16T17:07:14.392 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /dev 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /run 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /dev 2024-01-16T17:07:14.393 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:14.810 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-16T17:07:14.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-16T17:07:14.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-16T17:07:14.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-16T17:07:14.810 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-16T17:07:14.811 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-16T17:07:14.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-16T17:07:14.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-16T17:07:14.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-16T17:07:14.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-16T17:07:14.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-16T17:07:14.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-16T17:07:14.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-16T17:07:14.812 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-16T17:07:15.801 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-16T17:07:15.859 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-16T17:07:15.859 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-16T17:07:16.046 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-16T17:07:16.046 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.080 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: leveldb-1.22-1.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-16T17:07:16.081 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:16.082 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:07:16.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-16T17:07:16.733 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-16T17:07:16.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-16T17:07:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-16T17:07:17.104 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:43 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:07:18.074 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-16T17:07:18.074 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:18.074 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-01-16T17:07:18.074 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.074 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:07:18.074 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: leveldb-1.22-1.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-16T17:07:18.075 INFO:teuthology.orchestra.run.smithi190.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:18.076 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:07:18.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-01-16T17:07:18.855 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-16T17:07:18.856 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:18.857 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:07:18.857 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:18.857 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-01-16T17:07:18.857 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:18.857 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 53 M 2024-01-16T17:07:18.857 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 229 M 2024-01-16T17:07:18.857 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:07:19.155 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:43 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:07:19.192 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 169 kB 00:00 2024-01-16T17:07:19.801 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 73 MB/s | 52 MB 00:00 2024-01-16T17:07:19.826 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 275 kB/s | 203 kB 00:00 2024-01-16T17:07:19.851 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 285 kB/s | 188 kB 00:00 2024-01-16T17:07:20.018 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 368 kB/s | 71 kB 00:00 2024-01-16T17:07:20.351 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 587 kB/s | 323 kB 00:00 2024-01-16T17:07:20.352 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:07:20.353 INFO:teuthology.orchestra.run.smithi154.stdout:Total 35 MB/s | 53 MB 00:01 2024-01-16T17:07:20.396 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:07:20.418 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:07:20.418 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:07:20.730 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:07:20.733 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:07:20.878 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:07:20.879 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:20.879 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:07:20.879 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:20.879 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:07:20.879 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-16T17:07:20.879 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:07:20.879 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-16T17:07:20.879 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-16T17:07:20.880 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-16T17:07:20.880 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-16T17:07:20.880 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-16T17:07:20.880 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:20.880 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:07:20.880 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:20.880 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-01-16T17:07:20.880 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:20.881 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 53 M 2024-01-16T17:07:20.881 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 229 M 2024-01-16T17:07:20.881 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:07:21.051 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 169 kB 00:00 2024-01-16T17:07:21.247 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:07:21.429 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-16T17:07:21.612 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-16T17:07:21.727 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 71 MB/s | 52 MB 00:00 2024-01-16T17:07:21.752 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 267 kB/s | 203 kB 00:00 2024-01-16T17:07:21.777 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 259 kB/s | 188 kB 00:00 2024-01-16T17:07:21.810 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-16T17:07:21.869 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 604 kB/s | 71 kB 00:00 2024-01-16T17:07:21.873 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-16T17:07:22.059 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-16T17:07:22.102 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 861 kB/s | 323 kB 00:00 2024-01-16T17:07:22.104 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:07:22.104 INFO:teuthology.orchestra.run.smithi190.stdout:Total 43 MB/s | 53 MB 00:01 2024-01-16T17:07:22.147 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:07:22.168 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:07:22.168 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:07:22.218 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-16T17:07:22.487 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:07:22.490 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:07:22.968 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:07:23.203 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-16T17:07:23.431 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-16T17:07:23.617 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-16T17:07:23.715 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-16T17:07:23.899 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-16T17:07:24.084 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-16T17:07:27.127 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-16T17:07:27.390 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-16T17:07:27.700 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-16T17:07:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-16T17:07:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-16T17:07:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-16T17:07:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-16T17:07:27.701 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-16T17:07:28.477 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-16T17:07:28.477 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:28.477 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:07:28.477 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-16T17:07:28.477 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-16T17:07:28.477 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-16T17:07:28.477 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:28.477 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:07:28.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph 2024-01-16T17:07:28.931 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-16T17:07:29.198 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-16T17:07:29.484 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-16T17:07:29.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-16T17:07:29.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-16T17:07:29.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-16T17:07:29.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-16T17:07:29.484 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-16T17:07:29.561 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:55 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:07:30.221 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-16T17:07:30.221 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:30.221 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:07:30.221 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-16T17:07:30.221 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-16T17:07:30.221 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-16T17:07:30.221 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:30.221 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:07:30.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-01-16T17:07:31.300 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:07:31.303 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:31.303 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-01-16T17:07:31.303 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:31.303 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:07:31.303 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-16T17:07:31.303 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:07:31.303 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-16T17:07:31.303 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-16T17:07:31.304 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-16T17:07:31.305 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:31.306 INFO:teuthology.orchestra.run.smithi154.stdout:Install 32 Packages 2024-01-16T17:07:31.307 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:31.307 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 31 M 2024-01-16T17:07:31.307 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 111 M 2024-01-16T17:07:31.307 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:07:31.342 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:55 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:07:31.795 INFO:teuthology.orchestra.run.smithi154.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 112 kB/s | 6.3 kB 00:00 2024-01-16T17:07:31.845 INFO:teuthology.orchestra.run.smithi154.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-16T17:07:31.912 INFO:teuthology.orchestra.run.smithi154.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-01-16T17:07:31.937 INFO:teuthology.orchestra.run.smithi154.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-01-16T17:07:32.021 INFO:teuthology.orchestra.run.smithi154.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-01-16T17:07:32.246 INFO:teuthology.orchestra.run.smithi154.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-01-16T17:07:32.363 INFO:teuthology.orchestra.run.smithi154.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 369 kB/s | 157 kB 00:00 2024-01-16T17:07:32.538 INFO:teuthology.orchestra.run.smithi154.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 326 kB/s | 57 kB 00:00 2024-01-16T17:07:32.639 INFO:teuthology.orchestra.run.smithi154.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 264 kB/s | 103 kB 00:00 2024-01-16T17:07:32.664 INFO:teuthology.orchestra.run.smithi154.stdout:(10/32): python3-markupsafe-0.23-19.el8.x86_64. 60 kB/s | 39 kB 00:00 2024-01-16T17:07:32.739 INFO:teuthology.orchestra.run.smithi154.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2024-01-16T17:07:32.898 INFO:teuthology.orchestra.run.smithi154.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 961 kB/s | 249 kB 00:00 2024-01-16T17:07:32.948 INFO:teuthology.orchestra.run.smithi154.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 523 kB/s | 109 kB 00:00 2024-01-16T17:07:33.094 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:07:33.098 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:33.098 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-01-16T17:07:33.098 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:33.098 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:07:33.098 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-16T17:07:33.098 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:07:33.098 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-16T17:07:33.098 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-16T17:07:33.099 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-16T17:07:33.100 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout:Install 32 Packages 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 31 M 2024-01-16T17:07:33.101 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 111 M 2024-01-16T17:07:33.102 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:07:33.132 INFO:teuthology.orchestra.run.smithi154.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.3 MB/s | 641 kB 00:00 2024-01-16T17:07:33.282 INFO:teuthology.orchestra.run.smithi154.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 114 kB/s | 44 kB 00:00 2024-01-16T17:07:33.838 INFO:teuthology.orchestra.run.smithi190.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 104 kB/s | 6.3 kB 00:00 2024-01-16T17:07:33.897 INFO:teuthology.orchestra.run.smithi190.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-01-16T17:07:33.956 INFO:teuthology.orchestra.run.smithi190.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-01-16T17:07:33.989 INFO:teuthology.orchestra.run.smithi190.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-01-16T17:07:34.073 INFO:teuthology.orchestra.run.smithi190.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-01-16T17:07:34.290 INFO:teuthology.orchestra.run.smithi190.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 46 MB/s | 18 MB 00:00 2024-01-16T17:07:34.465 INFO:teuthology.orchestra.run.smithi190.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 330 kB/s | 157 kB 00:00 2024-01-16T17:07:34.491 INFO:teuthology.orchestra.run.smithi190.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 93 kB/s | 39 kB 00:00 2024-01-16T17:07:34.559 INFO:teuthology.orchestra.run.smithi190.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 619 kB/s | 57 kB 00:00 2024-01-16T17:07:34.590 INFO:teuthology.orchestra.run.smithi154.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 113 kB/s | 185 kB 00:01 2024-01-16T17:07:34.616 INFO:teuthology.orchestra.run.smithi190.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 317 kB/s | 103 kB 00:00 2024-01-16T17:07:34.649 INFO:teuthology.orchestra.run.smithi154.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 114 kB/s | 173 kB 00:01 2024-01-16T17:07:34.682 INFO:teuthology.orchestra.run.smithi154.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2024-01-16T17:07:34.717 INFO:teuthology.orchestra.run.smithi190.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.5 MB/s | 249 kB 00:00 2024-01-16T17:07:34.758 INFO:teuthology.orchestra.run.smithi154.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-01-16T17:07:34.758 INFO:teuthology.orchestra.run.smithi190.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.7 MB/s | 457 kB 00:00 2024-01-16T17:07:34.825 INFO:teuthology.orchestra.run.smithi190.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2024-01-16T17:07:34.966 INFO:teuthology.orchestra.run.smithi154.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 228 kB/s | 384 kB 00:01 2024-01-16T17:07:35.042 INFO:teuthology.orchestra.run.smithi190.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.5 MB/s | 641 kB 00:00 2024-01-16T17:07:35.091 INFO:teuthology.orchestra.run.smithi154.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 119 kB/s | 49 kB 00:00 2024-01-16T17:07:35.125 INFO:teuthology.orchestra.run.smithi154.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 160 kB/s | 59 kB 00:00 2024-01-16T17:07:35.184 INFO:teuthology.orchestra.run.smithi190.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 102 kB/s | 44 kB 00:00 2024-01-16T17:07:35.217 INFO:teuthology.orchestra.run.smithi154.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2024-01-16T17:07:35.326 INFO:teuthology.orchestra.run.smithi154.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 93 kB/s | 19 kB 00:00 2024-01-16T17:07:35.351 INFO:teuthology.orchestra.run.smithi154.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 179 kB/s | 24 kB 00:00 2024-01-16T17:07:35.510 INFO:teuthology.orchestra.run.smithi190.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 271 kB/s | 185 kB 00:00 2024-01-16T17:07:35.560 INFO:teuthology.orchestra.run.smithi154.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 125 kB/s | 29 kB 00:00 2024-01-16T17:07:35.635 INFO:teuthology.orchestra.run.smithi190.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 854 kB/s | 384 kB 00:00 2024-01-16T17:07:35.652 INFO:teuthology.orchestra.run.smithi154.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 92 kB/s | 27 kB 00:00 2024-01-16T17:07:35.652 INFO:teuthology.orchestra.run.smithi190.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2024-01-16T17:07:35.677 INFO:teuthology.orchestra.run.smithi190.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 272 kB/s | 173 kB 00:00 2024-01-16T17:07:35.744 INFO:teuthology.orchestra.run.smithi190.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-01-16T17:07:35.786 INFO:teuthology.orchestra.run.smithi190.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 364 kB/s | 49 kB 00:00 2024-01-16T17:07:35.819 INFO:teuthology.orchestra.run.smithi190.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 413 kB/s | 59 kB 00:00 2024-01-16T17:07:35.911 INFO:teuthology.orchestra.run.smithi190.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 124 kB/s | 16 kB 00:00 2024-01-16T17:07:35.937 INFO:teuthology.orchestra.run.smithi190.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 160 kB/s | 19 kB 00:00 2024-01-16T17:07:35.977 INFO:teuthology.orchestra.run.smithi154.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 280 kB/s | 283 kB 00:01 2024-01-16T17:07:36.004 INFO:teuthology.orchestra.run.smithi190.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2024-01-16T17:07:36.071 INFO:teuthology.orchestra.run.smithi190.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 150 kB/s | 24 kB 00:00 2024-01-16T17:07:36.096 INFO:teuthology.orchestra.run.smithi190.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 184 kB/s | 29 kB 00:00 2024-01-16T17:07:36.130 INFO:teuthology.orchestra.run.smithi190.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2024-01-16T17:07:36.280 INFO:teuthology.orchestra.run.smithi190.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 569 kB/s | 85 kB 00:00 2024-01-16T17:07:36.355 INFO:teuthology.orchestra.run.smithi190.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 898 kB/s | 255 kB 00:00 2024-01-16T17:07:36.381 INFO:teuthology.orchestra.run.smithi190.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 884 kB/s | 251 kB 00:00 2024-01-16T17:07:36.406 INFO:teuthology.orchestra.run.smithi190.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2024-01-16T17:07:36.407 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:07:36.407 INFO:teuthology.orchestra.run.smithi190.stdout:Total 9.4 MB/s | 31 MB 00:03 2024-01-16T17:07:36.493 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:07:36.531 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:07:36.531 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:07:36.861 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:07:36.864 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:07:36.869 INFO:teuthology.orchestra.run.smithi154.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 96 kB/s | 85 kB 00:00 2024-01-16T17:07:37.002 INFO:teuthology.orchestra.run.smithi154.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 177 kB/s | 255 kB 00:01 2024-01-16T17:07:37.061 INFO:teuthology.orchestra.run.smithi154.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 120 kB/s | 23 kB 00:00 2024-01-16T17:07:37.694 INFO:teuthology.orchestra.run.smithi154.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 123 kB/s | 251 kB 00:02 2024-01-16T17:07:37.699 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:07:37.699 INFO:teuthology.orchestra.run.smithi154.stdout:Total 4.8 MB/s | 31 MB 00:06 2024-01-16T17:07:37.783 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:07:37.822 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:07:37.822 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:07:37.970 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:07:38.137 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:07:38.140 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:07:38.194 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-16T17:07:38.465 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-16T17:07:38.706 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-16T17:07:38.954 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-16T17:07:39.127 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-16T17:07:39.240 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:07:39.316 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-16T17:07:39.454 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-16T17:07:39.534 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-16T17:07:39.712 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-16T17:07:39.712 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-16T17:07:39.905 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-16T17:07:39.965 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-16T17:07:40.142 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-16T17:07:40.154 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-16T17:07:40.311 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-16T17:07:40.371 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-16T17:07:40.457 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-16T17:07:40.603 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-16T17:07:40.667 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-16T17:07:40.828 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-16T17:07:41.011 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-16T17:07:41.021 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-16T17:07:41.206 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-16T17:07:41.317 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-16T17:07:41.428 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-16T17:07:41.480 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-16T17:07:41.676 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-16T17:07:41.713 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-16T17:07:41.923 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T17:07:42.028 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-16T17:07:42.205 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-16T17:07:42.275 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-16T17:07:42.459 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-16T17:07:42.480 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-16T17:07:42.658 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-16T17:07:42.706 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-16T17:07:42.869 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-16T17:07:42.907 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T17:07:43.178 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-16T17:07:43.200 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-16T17:07:43.409 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-16T17:07:43.472 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-16T17:07:43.641 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-16T17:07:43.682 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-16T17:07:43.827 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-16T17:07:44.012 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-16T17:07:44.048 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-16T17:07:44.310 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-16T17:07:44.361 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-16T17:07:44.527 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-16T17:07:44.741 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-16T17:07:44.827 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-16T17:07:44.954 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-16T17:07:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-16T17:07:45.530 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-16T17:07:45.742 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-16T17:07:46.485 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-16T17:07:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-16T17:07:47.110 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-16T17:07:47.320 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-16T17:07:47.342 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-16T17:07:47.541 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-16T17:07:47.764 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-16T17:07:47.782 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-16T17:07:47.841 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-16T17:07:47.913 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-16T17:07:48.279 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-16T17:07:48.315 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-16T17:07:48.315 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-16T17:07:48.316 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-16T17:07:48.317 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-16T17:07:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-16T17:07:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-16T17:07:48.318 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-16T17:07:48.483 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-16T17:07:48.706 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-16T17:07:48.863 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-16T17:07:48.882 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-16T17:07:48.996 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-16T17:07:49.412 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-16T17:07:49.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-16T17:07:49.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-16T17:07:49.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-16T17:07:49.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-16T17:07:49.412 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-16T17:07:49.413 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-16T17:07:49.414 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-16T17:07:50.370 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-16T17:07:50.371 INFO:teuthology.orchestra.run.smithi190.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-16T17:07:50.372 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-2.6-1.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-16T17:07:50.373 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:50.374 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-16T17:07:50.610 INFO:teuthology.orchestra.run.smithi154.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-2.6-1.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-16T17:07:50.611 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:50.612 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:07:50.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-01-16T17:07:50.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base 2024-01-16T17:07:51.570 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:15 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:07:51.620 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:17 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:07:53.057 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:07:53.122 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:07:53.301 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:07:53.302 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:07:53.302 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:07:53.366 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:07:53.367 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:07:53.367 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:07:53.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-01-16T17:07:53.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install cephadm 2024-01-16T17:07:54.301 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:18 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:07:54.331 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:20 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:07:56.050 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-01-16T17:07:56.051 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:56.052 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 71 k 2024-01-16T17:07:56.052 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 290 k 2024-01-16T17:07:56.052 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:07:56.079 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:07:56.080 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:56.080 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:07:56.080 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:56.080 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:07:56.080 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-16T17:07:56.080 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:56.080 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:07:56.081 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:07:56.081 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-01-16T17:07:56.081 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:56.081 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 71 k 2024-01-16T17:07:56.081 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 290 k 2024-01-16T17:07:56.081 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:07:56.095 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2024-01-16T17:07:56.096 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:07:56.096 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.6 MB/s | 71 kB 00:00 2024-01-16T17:07:56.097 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:07:56.103 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:07:56.103 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:07:56.105 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:07:56.108 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:07:56.132 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2024-01-16T17:07:56.133 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:07:56.133 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.3 MB/s | 71 kB 00:00 2024-01-16T17:07:56.133 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:07:56.140 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:07:56.140 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:07:56.142 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:07:56.145 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:07:56.152 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:07:56.184 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:07:56.634 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-16T17:07:56.667 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-16T17:07:56.853 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-16T17:07:56.906 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-16T17:07:57.083 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-16T17:07:57.140 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-16T17:07:57.802 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-16T17:07:57.802 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:57.802 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:07:57.802 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-16T17:07:57.803 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:07:57.803 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:07:57.895 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-16T17:07:57.895 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:57.895 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:07:57.896 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-16T17:07:57.896 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:07:57.896 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:07:58.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache 2024-01-16T17:07:58.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-01-16T17:07:58.913 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:24 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:07:58.996 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:23 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:00.682 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:00.682 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:00.682 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-01-16T17:08:00.682 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:00.682 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:08:00.682 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-16T17:08:00.682 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:00.683 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:08:00.683 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:00.683 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-01-16T17:08:00.683 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:00.683 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 163 k 2024-01-16T17:08:00.683 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 587 k 2024-01-16T17:08:00.683 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:08:00.728 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.5 MB/s | 163 kB 00:00 2024-01-16T17:08:00.729 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:00.729 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.4 MB/s | 163 kB 00:00 2024-01-16T17:08:00.729 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:08:00.740 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:08:00.740 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:08:00.744 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:00.745 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:00.745 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-01-16T17:08:00.745 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:00.745 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:08:00.745 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-16T17:08:00.745 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:00.746 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:08:00.746 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:00.746 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-01-16T17:08:00.746 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:00.746 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 163 k 2024-01-16T17:08:00.746 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 587 k 2024-01-16T17:08:00.746 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:08:00.788 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:08:00.791 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:08:00.898 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:08:01.039 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 556 kB/s | 163 kB 00:00 2024-01-16T17:08:01.040 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:01.040 INFO:teuthology.orchestra.run.smithi190.stdout:Total 554 kB/s | 163 kB 00:00 2024-01-16T17:08:01.041 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:08:01.052 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:08:01.052 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:08:01.061 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:01.100 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:08:01.103 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:08:01.473 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:01.683 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:08:02.173 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:02.173 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:02.173 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:08:02.173 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-16T17:08:02.173 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:02.173 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:02.228 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:02.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr 2024-01-16T17:08:02.645 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:03.181 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:29 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:03.384 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:03.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:03.384 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:08:03.384 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-16T17:08:03.384 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:03.384 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:03.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-01-16T17:08:04.451 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:28 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:04.680 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:08:04.925 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:04.926 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:08:04.926 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:05.037 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-dashboard 2024-01-16T17:08:05.897 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:31 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:05.969 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:08:06.212 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:06.213 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:08:06.213 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:06.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-01-16T17:08:07.174 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:31 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:07.627 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:07.628 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:07.628 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:08:07.629 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:07.630 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-01-16T17:08:07.630 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:07.630 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.0 M 2024-01-16T17:08:07.630 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 15 M 2024-01-16T17:08:07.630 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:08:07.915 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 430 kB/s | 24 kB 00:00 2024-01-16T17:08:07.940 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 110 kB/s | 8.9 kB 00:00 2024-01-16T17:08:07.998 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2024-01-16T17:08:08.099 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 212 kB/s | 34 kB 00:00 2024-01-16T17:08:08.124 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2024-01-16T17:08:08.358 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 548 kB/s | 196 kB 00:00 2024-01-16T17:08:08.359 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:08.360 INFO:teuthology.orchestra.run.smithi154.stdout:Total 4.0 MB/s | 3.0 MB 00:00 2024-01-16T17:08:08.370 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:08:08.379 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:08:08.379 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:08:08.551 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:08:08.554 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:08:08.826 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:08:08.929 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-16T17:08:08.930 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-16T17:08:08.931 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-16T17:08:08.931 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-16T17:08:08.931 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:08.931 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:08:08.931 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:08.931 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-01-16T17:08:08.931 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:08.931 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.0 M 2024-01-16T17:08:08.932 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 15 M 2024-01-16T17:08:08.932 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:08:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-16T17:08:09.243 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-16T17:08:09.486 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-16T17:08:09.607 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-16T17:08:09.696 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 456 kB/s | 24 kB 00:00 2024-01-16T17:08:09.721 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 115 kB/s | 8.9 kB 00:00 2024-01-16T17:08:09.788 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2024-01-16T17:08:09.872 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 224 kB/s | 34 kB 00:00 2024-01-16T17:08:09.947 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 170 kB/s | 43 kB 00:00 2024-01-16T17:08:09.972 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2024-01-16T17:08:09.975 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:09.975 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.8 MB/s | 3.0 MB 00:01 2024-01-16T17:08:09.986 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:08:09.996 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:08:09.997 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:08:10.173 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:08:10.176 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:08:10.459 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:08:10.587 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-16T17:08:10.673 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-16T17:08:10.937 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-16T17:08:10.968 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-16T17:08:11.186 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-16T17:08:11.210 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-16T17:08:11.210 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-16T17:08:11.210 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-16T17:08:11.210 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-16T17:08:11.210 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-16T17:08:11.210 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-16T17:08:11.340 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-16T17:08:11.926 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-16T17:08:11.926 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:11.927 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:12.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-16T17:08:12.321 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-16T17:08:12.941 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:38 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:12.949 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-16T17:08:13.292 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-16T17:08:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-16T17:08:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-16T17:08:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-16T17:08:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-16T17:08:13.293 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-16T17:08:13.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-16T17:08:13.999 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:13.999 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:08:13.999 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-16T17:08:13.999 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-16T17:08:13.999 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-16T17:08:14.000 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-16T17:08:14.000 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-16T17:08:14.000 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-16T17:08:14.000 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:14.000 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:14.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-16T17:08:14.699 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-16T17:08:14.701 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-16T17:08:14.702 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-16T17:08:14.702 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-16T17:08:14.702 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-16T17:08:14.702 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:14.702 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:08:14.702 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================================ 2024-01-16T17:08:14.702 INFO:teuthology.orchestra.run.smithi154.stdout:Install 8 Packages 2024-01-16T17:08:14.702 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:14.703 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 37 M 2024-01-16T17:08:14.703 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 210 M 2024-01-16T17:08:14.703 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:08:14.989 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:39 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:15.046 INFO:teuthology.orchestra.run.smithi154.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 58 MB/s | 8.6 MB 00:00 2024-01-16T17:08:15.746 INFO:teuthology.orchestra.run.smithi154.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.7 MB/s | 4.9 MB 00:00 2024-01-16T17:08:15.821 INFO:teuthology.orchestra.run.smithi154.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.1 MB/s | 4.7 MB 00:00 2024-01-16T17:08:15.846 INFO:teuthology.orchestra.run.smithi154.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.2 MB/s | 225 kB 00:00 2024-01-16T17:08:15.913 INFO:teuthology.orchestra.run.smithi154.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.3 MB/s | 3.7 MB 00:00 2024-01-16T17:08:16.297 INFO:teuthology.orchestra.run.smithi154.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 450 kB/s | 172 kB 00:00 2024-01-16T17:08:16.734 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:16.735 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-01-16T17:08:16.735 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-01-16T17:08:16.735 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:08:16.736 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-01-16T17:08:16.737 INFO:teuthology.orchestra.run.smithi190.stdout:Install 8 Packages 2024-01-16T17:08:16.737 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:16.737 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 37 M 2024-01-16T17:08:16.737 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 210 M 2024-01-16T17:08:16.737 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:08:17.050 INFO:teuthology.orchestra.run.smithi154.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 12 MB/s | 14 MB 00:01 2024-01-16T17:08:17.145 INFO:teuthology.orchestra.run.smithi190.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 47 MB/s | 8.6 MB 00:00 2024-01-16T17:08:17.166 INFO:teuthology.orchestra.run.smithi154.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 489 kB/s | 645 kB 00:01 2024-01-16T17:08:17.167 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:17.167 INFO:teuthology.orchestra.run.smithi154.stdout:Total 15 MB/s | 37 MB 00:02 2024-01-16T17:08:17.404 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:08:17.422 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:08:17.422 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:08:17.721 INFO:teuthology.orchestra.run.smithi190.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.2 MB/s | 4.7 MB 00:00 2024-01-16T17:08:17.736 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:08:17.739 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:08:17.779 INFO:teuthology.orchestra.run.smithi190.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.9 MB/s | 4.9 MB 00:00 2024-01-16T17:08:17.805 INFO:teuthology.orchestra.run.smithi190.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.6 MB/s | 225 kB 00:00 2024-01-16T17:08:17.871 INFO:teuthology.orchestra.run.smithi190.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.1 MB/s | 3.7 MB 00:00 2024-01-16T17:08:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:08:18.491 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T17:08:18.613 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T17:08:18.623 INFO:teuthology.orchestra.run.smithi190.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 229 kB/s | 172 kB 00:00 2024-01-16T17:08:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T17:08:18.932 INFO:teuthology.orchestra.run.smithi190.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 12 MB/s | 14 MB 00:01 2024-01-16T17:08:18.998 INFO:teuthology.orchestra.run.smithi190.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 541 kB/s | 645 kB 00:01 2024-01-16T17:08:18.999 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:18.999 INFO:teuthology.orchestra.run.smithi190.stdout:Total 16 MB/s | 37 MB 00:02 2024-01-16T17:08:19.234 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:08:19.252 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:08:19.252 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:08:19.368 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T17:08:19.557 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T17:08:19.587 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:08:19.590 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:08:20.169 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T17:08:20.185 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:08:20.325 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T17:08:20.332 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T17:08:20.473 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-16T17:08:20.649 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T17:08:21.005 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T17:08:21.247 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-16T17:08:21.310 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T17:08:21.408 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T17:08:21.540 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T17:08:22.023 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-16T17:08:22.174 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T17:08:22.904 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-16T17:08:23.252 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T17:08:23.340 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T17:08:23.524 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T17:08:24.633 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-16T17:08:24.853 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-16T17:08:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-16T17:08:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-16T17:08:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-16T17:08:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-16T17:08:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-16T17:08:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T17:08:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-16T17:08:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-16T17:08:25.307 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-16T17:08:25.937 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-16T17:08:25.937 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:25.937 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:08:25.937 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:25.938 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:26.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-rook 2024-01-16T17:08:26.584 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-16T17:08:26.770 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-16T17:08:26.905 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:52 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:27.138 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-16T17:08:27.138 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-16T17:08:27.138 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-16T17:08:27.138 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-16T17:08:27.138 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-16T17:08:27.138 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-16T17:08:27.138 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-16T17:08:27.138 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-16T17:08:27.900 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:27.901 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:28.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-01-16T17:08:28.664 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:28.665 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:28.665 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-01-16T17:08:28.665 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:28.665 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-16T17:08:28.666 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-16T17:08:28.667 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-16T17:08:28.667 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-16T17:08:28.667 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:28.667 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:08:28.667 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:28.667 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2024-01-16T17:08:28.667 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:28.668 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 1.9 M 2024-01-16T17:08:28.668 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 24 M 2024-01-16T17:08:28.668 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:08:28.870 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:52 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:28.972 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 505 kB/s | 27 kB 00:00 2024-01-16T17:08:29.089 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 117 kB/s | 20 kB 00:00 2024-01-16T17:08:29.114 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 136 kB/s | 27 kB 00:00 2024-01-16T17:08:29.189 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 582 kB/s | 126 kB 00:00 2024-01-16T17:08:29.240 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 730 kB/s | 110 kB 00:00 2024-01-16T17:08:29.307 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 806 kB/s | 155 kB 00:00 2024-01-16T17:08:29.382 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 222 kB/s | 43 kB 00:00 2024-01-16T17:08:29.457 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 152 kB/s | 33 kB 00:00 2024-01-16T17:08:29.474 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 97 kB/s | 16 kB 00:00 2024-01-16T17:08:29.666 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 286 kB/s | 81 kB 00:00 2024-01-16T17:08:29.691 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 296 kB/s | 64 kB 00:00 2024-01-16T17:08:29.858 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 320 kB/s | 61 kB 00:00 2024-01-16T17:08:30.610 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-16T17:08:30.612 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-01-16T17:08:30.613 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:30.614 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 1.9 M 2024-01-16T17:08:30.614 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 24 M 2024-01-16T17:08:30.614 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:08:30.817 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 878 kB/s | 1.2 MB 00:01 2024-01-16T17:08:30.817 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:30.818 INFO:teuthology.orchestra.run.smithi154.stdout:Total 910 kB/s | 1.9 MB 00:02 2024-01-16T17:08:30.876 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:08:30.892 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:08:30.892 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:08:30.980 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:08:30.983 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:08:31.213 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 504 kB/s | 27 kB 00:00 2024-01-16T17:08:31.322 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 123 kB/s | 20 kB 00:00 2024-01-16T17:08:31.364 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 130 kB/s | 27 kB 00:00 2024-01-16T17:08:31.473 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:08:31.506 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 432 kB/s | 126 kB 00:00 2024-01-16T17:08:31.531 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 526 kB/s | 110 kB 00:00 2024-01-16T17:08:31.590 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 685 kB/s | 155 kB 00:00 2024-01-16T17:08:31.674 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 255 kB/s | 43 kB 00:00 2024-01-16T17:08:31.699 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-16T17:08:31.774 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 136 kB/s | 33 kB 00:00 2024-01-16T17:08:31.816 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 74 kB/s | 16 kB 00:00 2024-01-16T17:08:31.905 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-16T17:08:31.924 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 593 kB/s | 64 kB 00:00 2024-01-16T17:08:31.966 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-google-auth-1.1.1-10.el8.noarc 278 kB/s | 81 kB 00:00 2024-01-16T17:08:31.992 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 916 kB/s | 61 kB 00:00 2024-01-16T17:08:32.115 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-16T17:08:32.311 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-16T17:08:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-16T17:08:32.659 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.3 MB/s | 1.2 MB 00:00 2024-01-16T17:08:32.662 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:32.662 INFO:teuthology.orchestra.run.smithi190.stdout:Total 955 kB/s | 1.9 MB 00:02 2024-01-16T17:08:32.721 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:08:32.737 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:08:32.737 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:08:32.774 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-16T17:08:32.825 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:08:32.828 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:08:33.041 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-16T17:08:33.292 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-16T17:08:33.357 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:08:33.583 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-16T17:08:33.747 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-16T17:08:34.016 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-16T17:08:34.255 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-16T17:08:34.314 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-16T17:08:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-16T17:08:34.611 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-16T17:08:34.768 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-16T17:08:34.811 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-16T17:08:34.987 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-16T17:08:35.111 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-16T17:08:35.146 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-16T17:08:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-16T17:08:35.425 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-16T17:08:35.426 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-16T17:08:36.225 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-16T17:08:36.225 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:36.225 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:08:36.225 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-16T17:08:36.225 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-16T17:08:36.225 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-16T17:08:36.225 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:36.226 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:36.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-cephadm 2024-01-16T17:08:36.425 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-16T17:08:36.737 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-16T17:08:36.937 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-16T17:08:37.129 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-16T17:08:37.242 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:03 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:37.330 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-16T17:08:37.635 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-16T17:08:37.636 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-16T17:08:37.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-16T17:08:37.637 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-16T17:08:38.424 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-16T17:08:38.425 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-16T17:08:38.425 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-16T17:08:38.425 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-16T17:08:38.425 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-16T17:08:38.425 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:38.425 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:38.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-01-16T17:08:38.976 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:38.977 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:38.977 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-01-16T17:08:38.978 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:38.979 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 5.8 M 2024-01-16T17:08:38.979 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 24 M 2024-01-16T17:08:38.979 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:08:39.266 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 89 kB 00:00 2024-01-16T17:08:39.366 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 212 kB/s | 21 kB 00:00 2024-01-16T17:08:39.427 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:03 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:39.508 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.7 MB/s | 538 kB 00:00 2024-01-16T17:08:39.700 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.6 MB/s | 4.8 MB 00:00 2024-01-16T17:08:39.725 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 225 kB/s | 49 kB 00:00 2024-01-16T17:08:39.750 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 926 kB/s | 355 kB 00:00 2024-01-16T17:08:39.751 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:39.751 INFO:teuthology.orchestra.run.smithi154.stdout:Total 7.5 MB/s | 5.8 MB 00:00 2024-01-16T17:08:39.803 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:08:39.812 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:08:39.812 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:08:39.937 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:08:39.940 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:08:40.219 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:08:40.455 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-16T17:08:40.707 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-16T17:08:41.168 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-16T17:08:41.169 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-16T17:08:41.170 INFO:teuthology.orchestra.run.smithi190.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-16T17:08:41.170 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:41.170 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:08:41.170 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:41.170 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-01-16T17:08:41.170 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:41.170 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 5.8 M 2024-01-16T17:08:41.170 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 24 M 2024-01-16T17:08:41.171 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:08:41.449 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 89 kB 00:00 2024-01-16T17:08:41.649 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-16T17:08:41.708 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 82 kB/s | 21 kB 00:00 2024-01-16T17:08:41.741 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.5 MB/s | 538 kB 00:00 2024-01-16T17:08:41.925 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 9.0 MB/s | 4.8 MB 00:00 2024-01-16T17:08:42.009 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-16T17:08:42.092 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 139 kB/s | 49 kB 00:00 2024-01-16T17:08:42.117 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 869 kB/s | 355 kB 00:00 2024-01-16T17:08:42.118 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:42.119 INFO:teuthology.orchestra.run.smithi190.stdout:Total 6.1 MB/s | 5.8 MB 00:00 2024-01-16T17:08:42.171 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:08:42.181 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:08:42.181 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:08:42.225 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-16T17:08:42.284 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:08:42.287 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:08:42.428 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-16T17:08:42.587 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:08:42.715 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-16T17:08:42.715 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-16T17:08:42.715 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-16T17:08:42.715 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-16T17:08:42.715 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-16T17:08:42.715 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-16T17:08:42.860 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-16T17:08:43.181 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-16T17:08:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-16T17:08:43.445 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:43.445 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:08:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-16T17:08:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-16T17:08:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-16T17:08:43.446 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:43.446 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:43.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse 2024-01-16T17:08:44.101 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-16T17:08:44.474 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-16T17:08:44.691 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-16T17:08:44.744 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:10 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:44.928 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-16T17:08:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-16T17:08:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-16T17:08:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-16T17:08:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-16T17:08:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-16T17:08:45.204 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-16T17:08:46.012 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-16T17:08:46.012 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:46.012 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:08:46.012 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-16T17:08:46.012 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-16T17:08:46.012 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-16T17:08:46.012 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:46.013 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:46.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-01-16T17:08:46.502 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:46.502 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:46.503 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2024-01-16T17:08:46.504 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:46.504 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 968 k 2024-01-16T17:08:46.504 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 3.2 M 2024-01-16T17:08:46.504 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:08:46.645 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2024-01-16T17:08:46.861 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 299 kB/s | 83 kB 00:00 2024-01-16T17:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:46.862 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.6 MB/s | 968 kB 00:00 2024-01-16T17:08:46.872 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:08:46.893 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:08:46.978 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:08:46.982 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:08:46.997 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:11 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:47.191 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:08:47.452 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-16T17:08:47.653 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-16T17:08:48.051 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-16T17:08:48.052 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-16T17:08:48.746 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:08:48.747 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:48.748 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-01-16T17:08:48.748 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:48.748 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 968 k 2024-01-16T17:08:48.748 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 3.2 M 2024-01-16T17:08:48.748 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:08:48.800 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-16T17:08:48.800 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:48.800 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:08:48.800 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-16T17:08:48.800 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:48.801 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:49.012 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2024-01-16T17:08:49.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel 2024-01-16T17:08:49.204 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 341 kB/s | 83 kB 00:00 2024-01-16T17:08:49.204 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:49.205 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.1 MB/s | 968 kB 00:00 2024-01-16T17:08:49.215 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:08:49.236 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:08:49.236 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:08:49.323 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:08:49.327 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:08:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:08:49.795 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-16T17:08:49.927 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:15 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:50.019 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-16T17:08:50.425 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-16T17:08:50.425 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-16T17:08:51.203 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-16T17:08:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:51.204 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:08:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-16T17:08:51.204 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:51.204 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:51.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-01-16T17:08:51.691 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:51.691 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:51.691 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:08:51.691 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:51.691 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:08:51.692 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-16T17:08:51.692 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:51.692 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:08:51.692 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:51.692 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-01-16T17:08:51.692 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:51.692 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 131 k 2024-01-16T17:08:51.693 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 439 k 2024-01-16T17:08:51.693 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:08:51.739 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 131 kB 00:00 2024-01-16T17:08:51.740 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:51.740 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.7 MB/s | 131 kB 00:00 2024-01-16T17:08:51.740 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:08:51.746 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:08:51.746 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:08:51.795 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:08:51.798 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:08:51.917 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:08:52.134 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:52.271 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:16 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:52.431 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:53.161 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:53.161 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:53.161 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:08:53.161 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-16T17:08:53.161 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:53.161 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:53.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2 2024-01-16T17:08:54.013 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-01-16T17:08:54.014 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:54.015 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 131 k 2024-01-16T17:08:54.015 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 439 k 2024-01-16T17:08:54.015 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:08:54.066 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.5 MB/s | 131 kB 00:00 2024-01-16T17:08:54.067 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:54.068 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-01-16T17:08:54.068 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:08:54.073 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:08:54.073 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:08:54.124 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:08:54.127 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:08:54.226 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:20 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:54.236 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:08:54.459 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:54.785 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:55.552 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:55.553 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:08:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-16T17:08:55.553 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:08:55.553 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:55.732 INFO:teuthology.orchestra.run.smithi154.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:08:55.979 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:55.981 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:08:55.981 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:08:56.387 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs-devel 2024-01-16T17:08:56.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-01-16T17:08:57.248 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:23 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:08:57.788 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:21 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:08:58.992 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:08:58.993 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:58.993 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:08:58.993 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:58.993 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:08:58.993 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-16T17:08:58.993 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:58.994 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:08:58.994 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:08:58.994 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-01-16T17:08:58.994 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:08:58.994 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 24 k 2024-01-16T17:08:58.994 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 94 k 2024-01-16T17:08:58.994 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:08:59.038 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 557 kB/s | 24 kB 00:00 2024-01-16T17:08:59.038 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:08:59.039 INFO:teuthology.orchestra.run.smithi154.stdout:Total 541 kB/s | 24 kB 00:00 2024-01-16T17:08:59.039 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:08:59.042 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:08:59.042 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:08:59.044 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:08:59.047 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:08:59.099 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:08:59.290 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:59.322 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:08:59.566 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:08:59.568 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:08:59.569 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:08:59.570 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:08:59.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-01-16T17:09:00.274 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:00.274 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:00.274 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:09:00.274 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-16T17:09:00.274 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:00.274 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:00.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2 2024-01-16T17:09:00.530 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:24 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:01.254 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:27 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:02.271 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:02.271 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:02.271 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:09:02.271 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:02.272 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:09:02.272 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-16T17:09:02.272 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:02.272 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:09:02.272 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:02.272 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-01-16T17:09:02.272 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:02.273 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 24 k 2024-01-16T17:09:02.273 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 94 k 2024-01-16T17:09:02.273 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:09:02.316 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 563 kB/s | 24 kB 00:00 2024-01-16T17:09:02.316 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:09:02.316 INFO:teuthology.orchestra.run.smithi190.stdout:Total 548 kB/s | 24 kB 00:00 2024-01-16T17:09:02.317 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:09:02.320 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:09:02.320 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:09:02.322 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:09:02.325 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:09:02.376 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:09:02.558 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:02.754 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:02.838 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:02.999 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:03.000 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:09:03.000 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:03.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1 2024-01-16T17:09:03.524 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:03.524 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:03.524 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:09:03.524 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-16T17:09:03.524 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:03.524 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:03.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-01-16T17:09:03.965 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:29 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:04.507 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:28 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:05.469 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:05.715 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:05.716 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:09:05.716 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:05.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados 2024-01-16T17:09:06.012 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:06.255 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:06.256 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:09:06.256 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:06.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-01-16T17:09:06.700 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:32 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:07.221 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:31 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:08.200 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:08.446 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:08.447 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:09:08.448 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:08.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw 2024-01-16T17:09:08.724 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:08.967 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:08.969 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:09:08.969 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:09.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-01-16T17:09:09.427 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:35 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:09.969 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:33 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:10.931 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:11.178 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:11.179 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:09:11.180 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:11.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs 2024-01-16T17:09:11.459 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:11.707 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:11.708 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:09:11.708 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:11.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-01-16T17:09:12.193 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:38 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:12.667 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:36 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:13.703 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:13.950 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:13.951 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:09:13.952 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:14.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd 2024-01-16T17:09:14.173 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:14.421 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:14.422 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:09:14.422 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:14.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-01-16T17:09:14.923 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:40 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:15.539 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:39 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:16.420 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:16.666 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:16.668 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-01-16T17:09:16.668 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:16.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse 2024-01-16T17:09:17.021 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:17.268 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:17.269 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:09:17.269 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:17.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-01-16T17:09:17.637 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:43 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:18.233 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:42 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:19.381 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:19.382 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:19.382 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:09:19.382 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:19.382 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:09:19.382 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-16T17:09:19.382 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:19.383 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:09:19.383 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:19.383 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-01-16T17:09:19.383 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:19.383 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 89 k 2024-01-16T17:09:19.383 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 298 k 2024-01-16T17:09:19.383 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:09:19.435 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 89 kB 00:00 2024-01-16T17:09:19.435 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:09:19.435 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-01-16T17:09:19.436 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:09:19.445 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:09:19.445 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:09:19.495 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:09:19.498 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:09:19.598 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:09:19.743 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-16T17:09:19.770 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:19.989 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:19.990 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-01-16T17:09:19.990 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:20.057 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:20.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-01-16T17:09:20.765 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:20.766 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:09:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-16T17:09:20.766 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:20.766 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:20.963 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:44 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:20.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror 2024-01-16T17:09:21.844 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:47 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:22.710 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-01-16T17:09:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:22.712 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 89 k 2024-01-16T17:09:22.712 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 298 k 2024-01-16T17:09:22.712 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:09:22.749 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 89 kB 00:00 2024-01-16T17:09:22.750 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:09:22.750 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.3 MB/s | 89 kB 00:00 2024-01-16T17:09:22.750 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:09:22.759 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:09:22.760 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:09:22.808 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:09:22.811 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:09:22.916 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:09:23.163 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:23.452 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:23.607 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:23.607 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:23.607 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:09:23.607 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:23.608 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:09:23.608 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-16T17:09:23.608 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:23.608 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:09:23.608 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:23.608 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-01-16T17:09:23.608 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:23.609 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.7 M 2024-01-16T17:09:23.609 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 18 M 2024-01-16T17:09:23.609 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:09:23.699 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-01-16T17:09:23.699 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:09:23.699 INFO:teuthology.orchestra.run.smithi154.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-01-16T17:09:23.699 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:09:23.711 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:09:23.711 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:09:23.774 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:09:23.777 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:09:24.168 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:24.181 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:24.181 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:09:24.181 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-16T17:09:24.182 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:24.182 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:24.249 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:09:24.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-01-16T17:09:24.446 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:24.851 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:25.244 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:49 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:25.554 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:25.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:25.555 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:09:25.555 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-16T17:09:25.555 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:25.555 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:25.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd 2024-01-16T17:09:26.645 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:02:52 ago on Tue 16 Jan 2024 05:06:34 PM UTC. 2024-01-16T17:09:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:26.988 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:26.988 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:09:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:09:26.989 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-16T17:09:26.989 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:09:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:26.989 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-01-16T17:09:26.989 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:26.990 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.7 M 2024-01-16T17:09:26.990 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 18 M 2024-01-16T17:09:26.990 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:09:27.077 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-01-16T17:09:27.078 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:09:27.078 INFO:teuthology.orchestra.run.smithi190.stdout:Total 42 MB/s | 3.7 MB 00:00 2024-01-16T17:09:27.079 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:09:27.090 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:09:27.090 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:09:27.153 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:09:27.156 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:09:27.618 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:09:27.822 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:28.247 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:28.397 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-01-16T17:09:28.397 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:28.397 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-01-16T17:09:28.397 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:28.397 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-01-16T17:09:28.397 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-16T17:09:28.397 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:28.398 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-01-16T17:09:28.398 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-01-16T17:09:28.398 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-01-16T17:09:28.398 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:28.398 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 173 k 2024-01-16T17:09:28.398 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 649 k 2024-01-16T17:09:28.398 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-01-16T17:09:28.442 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.9 MB/s | 173 kB 00:00 2024-01-16T17:09:28.442 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:09:28.443 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.8 MB/s | 173 kB 00:00 2024-01-16T17:09:28.443 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-01-16T17:09:28.454 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-01-16T17:09:28.454 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-01-16T17:09:28.503 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-01-16T17:09:28.506 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-01-16T17:09:28.623 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-01-16T17:09:28.802 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:28.943 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:09:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-16T17:09:28.943 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:28.943 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:29.104 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:29.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-01-16T17:09:29.853 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:29.854 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:29.854 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-01-16T17:09:29.854 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-16T17:09:29.854 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-01-16T17:09:29.854 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-01-16T17:09:30.010 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:54 ago on Tue 16 Jan 2024 05:06:36 PM UTC. 2024-01-16T17:09:30.032 DEBUG:teuthology.parallel:result is None 2024-01-16T17:09:31.735 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:31.736 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 173 k 2024-01-16T17:09:31.737 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 649 k 2024-01-16T17:09:31.737 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-01-16T17:09:31.772 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.8 MB/s | 173 kB 00:00 2024-01-16T17:09:31.772 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-01-16T17:09:31.773 INFO:teuthology.orchestra.run.smithi190.stdout:Total 4.6 MB/s | 173 kB 00:00 2024-01-16T17:09:31.773 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-01-16T17:09:31.784 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-01-16T17:09:31.784 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-01-16T17:09:31.832 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-01-16T17:09:31.835 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-01-16T17:09:31.951 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-01-16T17:09:32.153 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:32.484 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-16T17:09:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:33.255 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-01-16T17:09:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-16T17:09:33.255 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:09:33.255 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-01-16T17:09:33.472 DEBUG:teuthology.parallel:result is None 2024-01-16T17:09:33.472 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-16T17:09:33.472 INFO:teuthology.packaging:ref: None 2024-01-16T17:09:33.472 INFO:teuthology.packaging:tag: v16.2.4 2024-01-16T17:09:33.472 INFO:teuthology.packaging:branch: None 2024-01-16T17:09:33.473 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T17:09:33.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-16T17:09:33.852 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T17:09:33.879 INFO:teuthology.orchestra.run.smithi154.stdout:16.2.4-0.el8 2024-01-16T17:09:33.880 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-16T17:09:33.880 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-16T17:09:33.881 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-16T17:09:33.882 INFO:teuthology.packaging:ref: None 2024-01-16T17:09:33.882 INFO:teuthology.packaging:tag: v16.2.4 2024-01-16T17:09:33.882 INFO:teuthology.packaging:branch: None 2024-01-16T17:09:33.882 INFO:teuthology.packaging:sha1: a70aebef70df6ec2d957347ccbfc484e0573dd5f 2024-01-16T17:09:33.882 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-16T17:09:34.017 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T17:09:34.039 INFO:teuthology.orchestra.run.smithi190.stdout:16.2.4-0.el8 2024-01-16T17:09:34.039 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-16T17:09:34.040 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-16T17:09:34.041 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T17:09:34.041 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:09:34.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T17:09:34.074 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-01-16T17:09:34.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T17:09:34.107 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T17:09:34.107 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:09:34.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T17:09:34.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T17:09:34.214 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-01-16T17:09:34.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T17:09:34.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T17:09:34.307 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T17:09:34.307 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:09:34.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T17:09:34.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T17:09:34.404 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-01-16T17:09:34.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T17:09:34.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T17:09:34.498 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T17:09:34.498 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:09:34.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T17:09:34.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T17:09:34.594 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-01-16T17:09:34.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T17:09:34.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T17:09:34.687 INFO:teuthology.run_tasks:Running task print... 2024-01-16T17:09:34.696 INFO:teuthology.task.print:**** done install task... 2024-01-16T17:09:34.696 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-16T17:09:34.762 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'a70aebef70df6ec2d957347ccbfc484e0573dd5f'} 2024-01-16T17:09:34.762 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-16T17:09:34.763 INFO:tasks.cephadm:Cluster fsid is 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:09:34.763 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-16T17:09:34.763 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-16T17:09:34.763 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi154': '172.21.15.154', 'mon.smithi190': '172.21.15.190'} 2024-01-16T17:09:34.763 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-16T17:09:34.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2024-01-16T17:09:34.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-01-16T17:09:34.819 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-16T17:09:34.819 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-16T17:09:35.214 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 16 17:09 /home/ubuntu/cephtest/cephadm 2024-01-16T17:09:35.214 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-16T17:09:35.393 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 16 17:09 /home/ubuntu/cephtest/cephadm 2024-01-16T17:09:35.393 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-16T17:09:35.415 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-16T17:09:35.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2024-01-16T17:09:35.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-01-16T17:09:35.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2024-01-16T17:09:35.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-01-16T17:09:35.582 INFO:tasks.cephadm:Writing seed config... 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-16T17:09:35.583 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-16T17:09:35.584 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-16T17:09:35.585 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-16T17:09:35.586 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:09:35.586 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-16T17:09:35.613 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 059f8940-b492-11ee-95ae-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-16T17:09:35.613 DEBUG:teuthology.orchestra.run.smithi154:mon.smithi154> sudo journalctl -f -n 0 -u ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154.service 2024-01-16T17:09:35.656 INFO:tasks.cephadm:Bootstrapping... 2024-01-16T17:09:35.656 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 059f8940-b492-11ee-95ae-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.154 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-16T17:09:35.683 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:-- Logs begin at Tue 2024-01-16 16:58:55 UTC. -- 2024-01-16T17:09:35.924 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-16T17:09:36.838 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: 4.6.1 2024-01-16T17:09:36.848 INFO:teuthology.orchestra.run.smithi154.stderr:Verifying podman|docker is present... 2024-01-16T17:09:36.848 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-16T17:09:36.871 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: 4.6.1 2024-01-16T17:09:36.886 INFO:teuthology.orchestra.run.smithi154.stderr:Verifying lvm2 is present... 2024-01-16T17:09:36.887 INFO:teuthology.orchestra.run.smithi154.stderr:Verifying time synchronization is in place... 2024-01-16T17:09:36.887 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-enabled chrony.service 2024-01-16T17:09:36.894 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-16T17:09:36.895 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-active chrony.service 2024-01-16T17:09:36.901 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: inactive 2024-01-16T17:09:36.901 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-16T17:09:36.907 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: enabled 2024-01-16T17:09:36.908 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-active chronyd.service 2024-01-16T17:09:36.913 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: active 2024-01-16T17:09:36.914 INFO:teuthology.orchestra.run.smithi154.stderr:Unit chronyd.service is enabled and running 2024-01-16T17:09:36.914 INFO:teuthology.orchestra.run.smithi154.stderr:Repeating the final host check... 2024-01-16T17:09:36.914 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-16T17:09:36.935 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: 4.6.1 2024-01-16T17:09:36.947 INFO:teuthology.orchestra.run.smithi154.stderr:podman|docker (/bin/podman) is present 2024-01-16T17:09:36.947 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl is present 2024-01-16T17:09:36.947 INFO:teuthology.orchestra.run.smithi154.stderr:lvcreate is present 2024-01-16T17:09:36.947 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-enabled chrony.service 2024-01-16T17:09:36.953 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-16T17:09:36.953 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-active chrony.service 2024-01-16T17:09:36.959 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: inactive 2024-01-16T17:09:36.959 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-16T17:09:36.965 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: enabled 2024-01-16T17:09:36.966 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-active chronyd.service 2024-01-16T17:09:36.971 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: active 2024-01-16T17:09:36.972 INFO:teuthology.orchestra.run.smithi154.stderr:Unit chronyd.service is enabled and running 2024-01-16T17:09:36.972 INFO:teuthology.orchestra.run.smithi154.stderr:Host looks OK 2024-01-16T17:09:36.972 INFO:teuthology.orchestra.run.smithi154.stderr:Cluster fsid: 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:09:36.972 INFO:teuthology.orchestra.run.smithi154.stderr:Acquiring lock 140191401626312 on /run/cephadm/059f8940-b492-11ee-95ae-87774f69a715.lock 2024-01-16T17:09:36.972 INFO:teuthology.orchestra.run.smithi154.stderr:Lock 140191401626312 acquired on /run/cephadm/059f8940-b492-11ee-95ae-87774f69a715.lock 2024-01-16T17:09:36.973 INFO:teuthology.orchestra.run.smithi154.stderr:Verifying IP 172.21.15.154 port 3300 ... 2024-01-16T17:09:36.973 INFO:teuthology.orchestra.run.smithi154.stderr:Verifying IP 172.21.15.154 port 6789 ... 2024-01-16T17:09:36.973 INFO:teuthology.orchestra.run.smithi154.stderr:Base mon IP is 172.21.15.154, final addrv is [v2:172.21.15.154:3300,v1:172.21.15.154:6789] 2024-01-16T17:09:36.973 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /sbin/ip route ls 2024-01-16T17:09:36.975 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.154 metric 100 2024-01-16T17:09:36.975 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.154 metric 100 2024-01-16T17:09:36.976 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /sbin/ip -6 route ls 2024-01-16T17:09:36.978 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-16T17:09:36.978 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-16T17:09:36.978 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1778sec hoplimit 64 pref medium 2024-01-16T17:09:36.979 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /sbin/ip -6 addr ls 2024-01-16T17:09:36.981 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-16T17:09:36.981 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-16T17:09:36.981 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-16T17:09:36.981 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-16T17:09:36.981 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7761/64 scope link 2024-01-16T17:09:36.981 INFO:teuthology.orchestra.run.smithi154.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-16T17:09:36.982 INFO:teuthology.orchestra.run.smithi154.stderr:Mon IP 172.21.15.154 is in CIDR network 172.21.0.0/20 2024-01-16T17:09:36.982 INFO:teuthology.orchestra.run.smithi154.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-16T17:09:36.983 INFO:teuthology.orchestra.run.smithi154.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-16T17:09:36.984 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-16T17:09:37.080 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-16T17:09:44.076 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: Getting image source signatures 2024-01-16T17:09:44.083 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-16T17:09:44.083 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-16T17:09:59.800 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-16T17:09:59.866 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: Writing manifest to image destination 2024-01-16T17:10:00.140 INFO:teuthology.orchestra.run.smithi154.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-16T17:10:00.158 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-16T17:10:00.660 INFO:teuthology.orchestra.run.smithi154.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-16T17:10:01.007 INFO:teuthology.orchestra.run.smithi154.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-16T17:10:01.007 INFO:teuthology.orchestra.run.smithi154.stderr:Extracting ceph user uid/gid from container image... 2024-01-16T17:10:01.007 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-16T17:10:01.300 INFO:teuthology.orchestra.run.smithi154.stderr:stat: 167 167 2024-01-16T17:10:01.624 INFO:teuthology.orchestra.run.smithi154.stderr:Creating initial keys... 2024-01-16T17:10:01.624 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-16T17:10:01.874 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-authtool: AQBpuKZl8fmwNBAAuNKMqUodKmAciDN/lyMBUg== 2024-01-16T17:10:02.221 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-16T17:10:02.490 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-authtool: AQBquKZluNdWHRAAN1EQnZWsKdqlrwAVwcLgRw== 2024-01-16T17:10:02.830 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-16T17:10:03.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-authtool: AQBruKZlqvaCBBAAO/iTr4HOjdVrbP3/P4gGcA== 2024-01-16T17:10:03.394 INFO:teuthology.orchestra.run.smithi154.stderr:Creating initial monmap... 2024-01-16T17:10:03.394 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpb6h4oe2q:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 059f8940-b492-11ee-95ae-87774f69a715 --addv smithi154 [v2:172.21.15.154:3300,v1:172.21.15.154:6789] /tmp/monmap 2024-01-16T17:10:03.665 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-16T17:10:03.666 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:03.666 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-16T17:10:03.982 INFO:teuthology.orchestra.run.smithi154.stderr:monmaptool for smithi154 [v2:172.21.15.154:3300,v1:172.21.15.154:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-16T17:10:03.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/monmaptool: set fsid to 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:03.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-16T17:10:03.982 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:10:03.982 INFO:teuthology.orchestra.run.smithi154.stderr:Creating mon... 2024-01-16T17:10:03.983 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/mon.smithi154:/var/lib/ceph/mon/ceph-smithi154:z -v /tmp/ceph-tmpsw0qm3ue:/tmp/keyring:z -v /tmp/ceph-tmpb6h4oe2q:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi154 --fsid 059f8940-b492-11ee-95ae-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-16T17:10:04.318 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.296+0000 7ff6a5ab8700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-16T17:10:04.318 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.297+0000 7ff6a5ab8700 1 imported monmap: 2024-01-16T17:10:04.318 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-16T17:10:04.318 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: fsid 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:04.318 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: last_changed 2024-01-16T17:10:03.667056+0000 2024-01-16T17:10:04.318 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: created 2024-01-16T17:10:03.667056+0000 2024-01-16T17:10:04.318 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-16T17:10:04.318 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.smithi154 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.297+0000 7ff6a5ab8700 0 /usr/bin/ceph-mon: set fsid to 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: RocksDB version: 6.8.1 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Compile date May 13 2021 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: DB SUMMARY 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi154/store.db dir, Total Num: 0, files: 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi154/store.db: 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.error_if_exists: 0 2024-01-16T17:10:04.319 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.create_if_missing: 1 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.paranoid_checks: 1 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.env: 0x55f6e39661c0 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.fs: Posix File System 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.info_log: 0x55f6e5dd2ec0 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.statistics: (nil) 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.use_fsync: 0 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_log_file_size: 0 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.allow_fallocate: 1 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-16T17:10:04.320 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.use_direct_reads: 0 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.db_log_dir: 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi154/store.db 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_subcompactions: 1 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_background_flushes: -1 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-16T17:10:04.321 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.write_buffer_manager: 0x55f6e5dd9530 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.unordered_write: 0 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.row_cache: None 2024-01-16T17:10:04.322 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.wal_filter: None 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.preserve_deletes: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.two_write_queues: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.atomic_flush: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.log_readahead_size: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_background_jobs: 2 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_background_compactions: -1 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-16T17:10:04.323 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_open_files: -1 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Compression algorithms supported: 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: kZSTD supported: 0 2024-01-16T17:10:04.324 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: kXpressCompression supported: 0 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: kLZ4Compression supported: 1 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: kBZip2Compression supported: 0 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: kZlibCompression supported: 1 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: kSnappyCompression supported: 1 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi154/store.db/MANIFEST-000001 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.merge_operator: 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_filter: None 2024-01-16T17:10:04.325 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f6e5cea120) 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: block_cache: 0x55f6e5d20d10 2024-01-16T17:10:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-16T17:10:04.327 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compression: NoCompression 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.num_levels: 7 2024-01-16T17:10:04.328 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-16T17:10:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-16T17:10:04.330 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-16T17:10:04.331 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.table_properties_collectors: 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.inplace_update_support: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.bloom_locality: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.max_successive_merges: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.ttl: 2592000 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.298+0000 7ff6a5ab8700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-16T17:10:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.299+0000 7ff6a5ab8700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi154/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.299+0000 7ff6a5ab8700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.299+0000 7ff6a5ab8700 4 rocksdb: DB pointer 0x55f6e5de5800 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.299+0000 7ff68e951700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.300+0000 7ff68e951700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-16T17:10:04.334 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-16T17:10:04.335 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.301+0000 7ff6a5ab8700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.301+0000 7ff6a5ab8700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-16T17:10:04.336 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph-mon: debug 2024-01-16T17:10:04.301+0000 7ff6a5ab8700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi154 for mon.smithi154 2024-01-16T17:10:04.627 INFO:teuthology.orchestra.run.smithi154.stderr:create mon.smithi154 on 2024-01-16T17:10:04.627 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:04.632 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl disable ceph.target 2024-01-16T17:10:04.639 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-16T17:10:04.723 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl enable ceph.target 2024-01-16T17:10:04.730 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-16T17:10:04.811 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl start ceph.target 2024-01-16T17:10:04.818 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl enable ceph-059f8940-b492-11ee-95ae-87774f69a715.target 2024-01-16T17:10:04.824 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-059f8940-b492-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-059f8940-b492-11ee-95ae-87774f69a715.target. 2024-01-16T17:10:04.825 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-059f8940-b492-11ee-95ae-87774f69a715.target → /etc/systemd/system/ceph-059f8940-b492-11ee-95ae-87774f69a715.target. 2024-01-16T17:10:04.904 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl start ceph-059f8940-b492-11ee-95ae-87774f69a715.target 2024-01-16T17:10:04.911 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl daemon-reload 2024-01-16T17:10:04.998 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154 2024-01-16T17:10:05.021 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl reset-failed ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154 2024-01-16T17:10:05.027 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Failed to reset failed state of unit ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154.service: Unit ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154.service not loaded. 2024-01-16T17:10:05.027 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl enable ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154 2024-01-16T17:10:05.034 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Created symlink /etc/systemd/system/ceph-059f8940-b492-11ee-95ae-87774f69a715.target.wants/ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154.service → /etc/systemd/system/ceph-059f8940-b492-11ee-95ae-87774f69a715@.service. 2024-01-16T17:10:05.117 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl start ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154 2024-01-16T17:10:05.403 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-16T17:10:05.412 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: disabled 2024-01-16T17:10:05.413 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-active firewalld.service 2024-01-16T17:10:05.420 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: inactive 2024-01-16T17:10:05.421 INFO:teuthology.orchestra.run.smithi154.stderr:firewalld.service is not enabled 2024-01-16T17:10:05.421 INFO:teuthology.orchestra.run.smithi154.stderr:Not possible to enable service . firewalld.service is not available 2024-01-16T17:10:05.421 INFO:teuthology.orchestra.run.smithi154.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-16T17:10:05.422 INFO:teuthology.orchestra.run.smithi154.stderr:Waiting for mon to start... 2024-01-16T17:10:05.422 INFO:teuthology.orchestra.run.smithi154.stderr:Waiting for mon... 2024-01-16T17:10:05.422 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/mon.smithi154:/var/lib/ceph/mon/ceph-smithi154:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-16T17:10:05.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: cluster 2024-01-16T17:10:05.438006+0000 mon.smithi154 (mon.0) 1 : cluster [INF] mon.smithi154 is new leader, mons smithi154 in quorum (ranks 0) 2024-01-16T17:10:05.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.813+0000 7f524359e700 1 Processor -- start 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.813+0000 7f524359e700 1 -- start start 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.813+0000 7f524359e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f52440fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.813+0000 7f524359e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52440fec60 con 0x7f52440703f0 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.813+0000 7f524259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f52440fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.813+0000 7f524259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f52440fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40152/0 (socket says 172.21.15.154:40152) 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.813+0000 7f524259c700 1 -- 172.21.15.154:0/3231570777 learned_addr learned my addr 172.21.15.154:0/3231570777 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.814+0000 7f524259c700 1 -- 172.21.15.154:0/3231570777 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52440feda0 con 0x7f52440703f0 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.814+0000 7f524259c700 1 --2- 172.21.15.154:0/3231570777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f52440fe770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f523400b5f0 tx=0x7f5234006ee0).ready entity=mon.0 client_cookie=f6dec3205a41d842 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.814+0000 7f524159a700 1 -- 172.21.15.154:0/3231570777 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f523400d600 con 0x7f52440703f0 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.814+0000 7f524159a700 1 -- 172.21.15.154:0/3231570777 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523400d760 con 0x7f52440703f0 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.814+0000 7f524159a700 1 -- 172.21.15.154:0/3231570777 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f523400da30 con 0x7f52440703f0 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.814+0000 7f524359e700 1 -- 172.21.15.154:0/3231570777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 msgr2=0x7f52440fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:05.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.815+0000 7f524359e700 1 --2- 172.21.15.154:0/3231570777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f52440fe770 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f523400b5f0 tx=0x7f5234006ee0).stop 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.815+0000 7f524359e700 1 -- 172.21.15.154:0/3231570777 shutdown_connections 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.815+0000 7f524359e700 1 --2- 172.21.15.154:0/3231570777 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f52440fe770 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.815+0000 7f524359e700 1 -- 172.21.15.154:0/3231570777 >> 172.21.15.154:0/3231570777 conn(0x7f52440f6210 msgr2=0x7f52440f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.815+0000 7f524359e700 1 -- 172.21.15.154:0/3231570777 shutdown_connections 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.815+0000 7f524359e700 1 -- 172.21.15.154:0/3231570777 wait complete. 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.815+0000 7f524359e700 1 Processor -- start 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.815+0000 7f524359e700 1 -- start start 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524359e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f524410a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524359e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524410a5f0 con 0x7f52440703f0 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f524410a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f524410a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40158/0 (socket says 172.21.15.154:40158) 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524259c700 1 -- 172.21.15.154:0/3229273279 learned_addr learned my addr 172.21.15.154:0/3229273279 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524259c700 1 -- 172.21.15.154:0/3229273279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523400b200 con 0x7f52440703f0 2024-01-16T17:10:05.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524259c700 1 --2- 172.21.15.154:0/3229273279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f524410a100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5234006c70 tx=0x7f5234016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f5232ffd700 1 -- 172.21.15.154:0/3229273279 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5234011b40 con 0x7f52440703f0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f5232ffd700 1 -- 172.21.15.154:0/3229273279 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234011ca0 con 0x7f52440703f0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524410a7f0 con 0x7f52440703f0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f5232ffd700 1 -- 172.21.15.154:0/3229273279 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f523401d430 con 0x7f52440703f0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.816+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524410ac00 con 0x7f52440703f0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.817+0000 7f5232ffd700 1 -- 172.21.15.154:0/3229273279 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f523400f070 con 0x7f52440703f0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.817+0000 7f5232ffd700 1 -- 172.21.15.154:0/3229273279 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5234027460 con 0x7f52440703f0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.817+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5244045ad0 con 0x7f52440703f0 2024-01-16T17:10:05.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.820+0000 7f5232ffd700 1 -- 172.21.15.154:0/3229273279 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5234016070 con 0x7f52440703f0 2024-01-16T17:10:05.870 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.885+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5244103cb0 con 0x7f52440703f0 2024-01-16T17:10:05.870 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.886+0000 7f5232ffd700 1 -- 172.21.15.154:0/3229273279 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+323 (secure 0 0 0) 0x7f5234036070 con 0x7f52440703f0 2024-01-16T17:10:05.870 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: cluster: 2024-01-16T17:10:05.870 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: id: 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:05.870 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-16T17:10:05.870 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.870 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: services: 2024-01-16T17:10:05.870 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi154 (age 0.4326s) 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: data: 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: pgs: 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.886+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 msgr2=0x7f524410a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.886+0000 7f524359e700 1 --2- 172.21.15.154:0/3229273279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f524410a100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5234006c70 tx=0x7f5234016000).stop 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.886+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 shutdown_connections 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.886+0000 7f524359e700 1 --2- 172.21.15.154:0/3229273279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52440703f0 0x7f524410a100 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.886+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 >> 172.21.15.154:0/3229273279 conn(0x7f52440f6210 msgr2=0x7f52440f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.886+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 shutdown_connections 2024-01-16T17:10:05.871 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:05.886+0000 7f524359e700 1 -- 172.21.15.154:0/3229273279 wait complete. 2024-01-16T17:10:06.235 INFO:teuthology.orchestra.run.smithi154.stderr:mon is available 2024-01-16T17:10:06.235 INFO:teuthology.orchestra.run.smithi154.stderr:Assimilating anything we can from ceph.conf... 2024-01-16T17:10:06.235 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z -v /var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/mon.smithi154:/var/lib/ceph/mon/ceph-smithi154:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi154/config 2024-01-16T17:10:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.594+0000 7f5d1e27c700 1 Processor -- start 2024-01-16T17:10:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.594+0000 7f5d1e27c700 1 -- start start 2024-01-16T17:10:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.594+0000 7f5d1e27c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d18100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.594+0000 7f5d1e27c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d180fc750 con 0x7f5d180703b0 2024-01-16T17:10:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.595+0000 7f5d17fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d18100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.595+0000 7f5d17fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d18100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40172/0 (socket says 172.21.15.154:40172) 2024-01-16T17:10:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.595+0000 7f5d17fff700 1 -- 172.21.15.154:0/2431054749 learned_addr learned my addr 172.21.15.154:0/2431054749 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:06.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.595+0000 7f5d17fff700 1 -- 172.21.15.154:0/2431054749 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d18101250 con 0x7f5d180703b0 2024-01-16T17:10:06.579 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.595+0000 7f5d17fff700 1 --2- 172.21.15.154:0/2431054749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d18100d60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f5d0c00b5f0 tx=0x7f5d0c006ee0).ready entity=mon.0 client_cookie=9fea84793b96e733 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:06.579 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.595+0000 7f5d16ffd700 1 -- 172.21.15.154:0/2431054749 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d0c00d600 con 0x7f5d180703b0 2024-01-16T17:10:06.579 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.595+0000 7f5d16ffd700 1 -- 172.21.15.154:0/2431054749 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c00d760 con 0x7f5d180703b0 2024-01-16T17:10:06.579 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.595+0000 7f5d16ffd700 1 -- 172.21.15.154:0/2431054749 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d0c00da30 con 0x7f5d180703b0 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.596+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2431054749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 msgr2=0x7f5d18100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.596+0000 7f5d1e27c700 1 --2- 172.21.15.154:0/2431054749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d18100d60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f5d0c00b5f0 tx=0x7f5d0c006ee0).stop 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.596+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2431054749 shutdown_connections 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.596+0000 7f5d1e27c700 1 --2- 172.21.15.154:0/2431054749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d18100d60 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.596+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2431054749 >> 172.21.15.154:0/2431054749 conn(0x7f5d180f63c0 msgr2=0x7f5d180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.596+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2431054749 shutdown_connections 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.596+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2431054749 wait complete. 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d1e27c700 1 Processor -- start 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d1e27c700 1 -- start start 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d1e27c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d1810c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d1e27c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d1810c970 con 0x7f5d180703b0 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d17fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d1810c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:06.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d17fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d1810c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40174/0 (socket says 172.21.15.154:40174) 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d17fff700 1 -- 172.21.15.154:0/2965030653 learned_addr learned my addr 172.21.15.154:0/2965030653 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d17fff700 1 -- 172.21.15.154:0/2965030653 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c00b200 con 0x7f5d180703b0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.597+0000 7f5d17fff700 1 --2- 172.21.15.154:0/2965030653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d1810c480 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f5d0c006c70 tx=0x7f5d0c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.598+0000 7f5d14ff9700 1 -- 172.21.15.154:0/2965030653 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d0c011b40 con 0x7f5d180703b0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.598+0000 7f5d14ff9700 1 -- 172.21.15.154:0/2965030653 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c011ca0 con 0x7f5d180703b0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.598+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d1810cb70 con 0x7f5d180703b0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.598+0000 7f5d14ff9700 1 -- 172.21.15.154:0/2965030653 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d0c01d430 con 0x7f5d180703b0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.598+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d1810cf80 con 0x7f5d180703b0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.598+0000 7f5d14ff9700 1 -- 172.21.15.154:0/2965030653 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5d0c00f070 con 0x7f5d180703b0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.598+0000 7f5d14ff9700 1 -- 172.21.15.154:0/2965030653 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5d0c027460 con 0x7f5d180703b0 2024-01-16T17:10:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.599+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d18045ad0 con 0x7f5d180703b0 2024-01-16T17:10:06.584 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.602+0000 7f5d14ff9700 1 -- 172.21.15.154:0/2965030653 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5d0c016070 con 0x7f5d180703b0 2024-01-16T17:10:06.647 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.665+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f5d181013e0 con 0x7f5d180703b0 2024-01-16T17:10:06.653 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.668+0000 7f5d14ff9700 1 -- 172.21.15.154:0/2965030653 <== mon.0 v2:172.21.15.154:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5d0c01d660 con 0x7f5d180703b0 2024-01-16T17:10:06.653 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.668+0000 7f5d14ff9700 1 -- 172.21.15.154:0/2965030653 <== mon.0 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7f5d0c036020 con 0x7f5d180703b0 2024-01-16T17:10:06.653 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.653 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: [global] 2024-01-16T17:10:06.653 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: fsid = 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:06.653 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.154:3300,v1:172.21.15.154:6789] 2024-01-16T17:10:06.653 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-16T17:10:06.653 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: [mgr] 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: [osd] 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.669+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 msgr2=0x7f5d1810c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.669+0000 7f5d1e27c700 1 --2- 172.21.15.154:0/2965030653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d1810c480 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f5d0c006c70 tx=0x7f5d0c016000).stop 2024-01-16T17:10:06.654 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.669+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 shutdown_connections 2024-01-16T17:10:06.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.669+0000 7f5d1e27c700 1 --2- 172.21.15.154:0/2965030653 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d180703b0 0x7f5d1810c480 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:06.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.669+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 >> 172.21.15.154:0/2965030653 conn(0x7f5d180f63c0 msgr2=0x7f5d180f8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:06.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.669+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 shutdown_connections 2024-01-16T17:10:06.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:06.669+0000 7f5d1e27c700 1 -- 172.21.15.154:0/2965030653 wait complete. 2024-01-16T17:10:06.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: cluster 2024-01-16T17:10:05.454652+0000 mon.smithi154 (mon.0) 2 : cluster [INF] mon.smithi154 is new leader, mons smithi154 in quorum (ranks 0) 2024-01-16T17:10:06.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: cluster 2024-01-16T17:10:05.454891+0000 mon.smithi154 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi154=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0]} 2024-01-16T17:10:06.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: cluster 2024-01-16T17:10:05.458117+0000 mon.smithi154 (mon.0) 4 : cluster [DBG] fsmap 2024-01-16T17:10:06.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: cluster 2024-01-16T17:10:05.460168+0000 mon.smithi154 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-16T17:10:06.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: cluster 2024-01-16T17:10:05.460568+0000 mon.smithi154 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-16T17:10:06.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: audit 2024-01-16T17:10:05.887212+0000 mon.smithi154 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.154:0/3229273279' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-16T17:10:07.009 INFO:teuthology.orchestra.run.smithi154.stderr:Generating new minimal ceph.conf... 2024-01-16T17:10:07.009 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z -v /var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/mon.smithi154:/var/lib/ceph/mon/ceph-smithi154:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi154/config 2024-01-16T17:10:07.387 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.401+0000 7fd4ef2a4700 1 Processor -- start 2024-01-16T17:10:07.387 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.401+0000 7fd4ef2a4700 1 -- start start 2024-01-16T17:10:07.387 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.401+0000 7fd4ef2a4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e80fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:07.387 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.401+0000 7fd4ef2a4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e80fec60 con 0x7fd4e80703f0 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.401+0000 7fd4ee2a2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e80fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.401+0000 7fd4ee2a2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e80fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40190/0 (socket says 172.21.15.154:40190) 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.401+0000 7fd4ee2a2700 1 -- 172.21.15.154:0/3020575205 learned_addr learned my addr 172.21.15.154:0/3020575205 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.402+0000 7fd4ee2a2700 1 -- 172.21.15.154:0/3020575205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80feda0 con 0x7fd4e80703f0 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.402+0000 7fd4ee2a2700 1 --2- 172.21.15.154:0/3020575205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e80fe770 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd4e400b5f0 tx=0x7fd4e4006ee0).ready entity=mon.0 client_cookie=f490629494355d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.402+0000 7fd4ed2a0700 1 -- 172.21.15.154:0/3020575205 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e400d600 con 0x7fd4e80703f0 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.402+0000 7fd4ed2a0700 1 -- 172.21.15.154:0/3020575205 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd4e400dbd0 con 0x7fd4e80703f0 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.402+0000 7fd4ed2a0700 1 -- 172.21.15.154:0/3020575205 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e4011c00 con 0x7fd4e80703f0 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.402+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/3020575205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 msgr2=0x7fd4e80fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.402+0000 7fd4ef2a4700 1 --2- 172.21.15.154:0/3020575205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e80fe770 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd4e400b5f0 tx=0x7fd4e4006ee0).stop 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/3020575205 shutdown_connections 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 --2- 172.21.15.154:0/3020575205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e80fe770 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/3020575205 >> 172.21.15.154:0/3020575205 conn(0x7fd4e80f6210 msgr2=0x7fd4e80f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/3020575205 shutdown_connections 2024-01-16T17:10:07.388 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/3020575205 wait complete. 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 Processor -- start 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 -- start start 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e810a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ef2a4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e810ad70 con 0x7fd4e80703f0 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ee2a2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e810a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.403+0000 7fd4ee2a2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e810a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40200/0 (socket says 172.21.15.154:40200) 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4ee2a2700 1 -- 172.21.15.154:0/352468969 learned_addr learned my addr 172.21.15.154:0/352468969 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4ee2a2700 1 -- 172.21.15.154:0/352468969 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e400b200 con 0x7fd4e80703f0 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4ee2a2700 1 --2- 172.21.15.154:0/352468969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e810a880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd4e400e040 tx=0x7fd4e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4d6ffd700 1 -- 172.21.15.154:0/352468969 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e400d9f0 con 0x7fd4e80703f0 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4d6ffd700 1 -- 172.21.15.154:0/352468969 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd4e401a9f0 con 0x7fd4e80703f0 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e810af70 con 0x7fd4e80703f0 2024-01-16T17:10:07.389 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4d6ffd700 1 -- 172.21.15.154:0/352468969 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4e401b8c0 con 0x7fd4e80703f0 2024-01-16T17:10:07.390 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e810b380 con 0x7fd4e80703f0 2024-01-16T17:10:07.390 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4d6ffd700 1 -- 172.21.15.154:0/352468969 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd4e400f070 con 0x7fd4e80703f0 2024-01-16T17:10:07.390 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.404+0000 7fd4d6ffd700 1 -- 172.21.15.154:0/352468969 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd4e4025b30 con 0x7fd4e80703f0 2024-01-16T17:10:07.390 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.405+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e8045ad0 con 0x7fd4e80703f0 2024-01-16T17:10:07.390 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.407+0000 7fd4d6ffd700 1 -- 172.21.15.154:0/352468969 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd4e4016070 con 0x7fd4e80703f0 2024-01-16T17:10:07.454 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.472+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd4e80ffa20 con 0x7fd4e80703f0 2024-01-16T17:10:07.454 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.472+0000 7fd4d6ffd700 1 -- 172.21.15.154:0/352468969 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fd4e401bd20 con 0x7fd4e80703f0 2024-01-16T17:10:07.456 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.473+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 msgr2=0x7fd4e810a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:07.456 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.473+0000 7fd4ef2a4700 1 --2- 172.21.15.154:0/352468969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e810a880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd4e400e040 tx=0x7fd4e4006ee0).stop 2024-01-16T17:10:07.456 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.473+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 shutdown_connections 2024-01-16T17:10:07.456 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.473+0000 7fd4ef2a4700 1 --2- 172.21.15.154:0/352468969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4e80703f0 0x7fd4e810a880 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:07.456 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.473+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 >> 172.21.15.154:0/352468969 conn(0x7fd4e80f6210 msgr2=0x7fd4e80f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:07.456 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.473+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 shutdown_connections 2024-01-16T17:10:07.456 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:07.473+0000 7fd4ef2a4700 1 -- 172.21.15.154:0/352468969 wait complete. 2024-01-16T17:10:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: audit 2024-01-16T17:10:06.666661+0000 mon.smithi154 (mon.0) 8 : audit [INF] from='client.? 172.21.15.154:0/2965030653' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-16T17:10:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: audit 2024-01-16T17:10:06.669725+0000 mon.smithi154 (mon.0) 9 : audit [INF] from='client.? 172.21.15.154:0/2965030653' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-16T17:10:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:Restarting the monitor... 2024-01-16T17:10:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl restart ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154 2024-01-16T17:10:08.006 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:07 smithi154 systemd[1]: Stopping Ceph mon.smithi154 for 059f8940-b492-11ee-95ae-87774f69a715... 2024-01-16T17:10:08.006 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: debug 2024-01-16T17:10:07.879+0000 7f7202a81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi154 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-16T17:10:08.006 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67115]: debug 2024-01-16T17:10:07.879+0000 7f7202a81700 -1 mon.smithi154@0(leader) e1 *** Got Signal Terminated *** 2024-01-16T17:10:08.290 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 podman[67437]: ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154 2024-01-16T17:10:08.290 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 systemd[1]: ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154.service: Succeeded. 2024-01-16T17:10:08.290 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 systemd[1]: Stopped Ceph mon.smithi154 for 059f8940-b492-11ee-95ae-87774f69a715. 2024-01-16T17:10:08.290 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 systemd[1]: Starting Ceph mon.smithi154 for 059f8940-b492-11ee-95ae-87774f69a715... 2024-01-16T17:10:08.342 INFO:teuthology.orchestra.run.smithi154.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-16T17:10:08.343 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.719+0000 7f2ac4a82700 1 Processor -- start 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.719+0000 7f2ac4a82700 1 -- start start 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.719+0000 7f2ac4a82700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac00fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.719+0000 7f2ac4a82700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac0070790 con 0x7f2ac0070370 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.720+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac00fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.720+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac00fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40208/0 (socket says 172.21.15.154:40208) 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.720+0000 7f2abe59c700 1 -- 172.21.15.154:0/1375955729 learned_addr learned my addr 172.21.15.154:0/1375955729 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.720+0000 7f2abe59c700 1 -- 172.21.15.154:0/1375955729 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac00fee90 con 0x7f2ac0070370 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.720+0000 7f2abe59c700 1 --2- 172.21.15.154:0/1375955729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac00fe9a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2ab000b5f0 tx=0x7f2ab0006ee0).ready entity=mon.0 client_cookie=c7586b9cdea38db4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.720+0000 7f2abd59a700 1 -- 172.21.15.154:0/1375955729 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab000d600 con 0x7f2ac0070370 2024-01-16T17:10:08.706 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.720+0000 7f2abd59a700 1 -- 172.21.15.154:0/1375955729 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2ab000dbd0 con 0x7f2ac0070370 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.720+0000 7f2abd59a700 1 -- 172.21.15.154:0/1375955729 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab0011c00 con 0x7f2ac0070370 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.721+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1375955729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 msgr2=0x7f2ac00fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.721+0000 7f2ac4a82700 1 --2- 172.21.15.154:0/1375955729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac00fe9a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f2ab000b5f0 tx=0x7f2ab0006ee0).stop 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.721+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1375955729 shutdown_connections 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.721+0000 7f2ac4a82700 1 --2- 172.21.15.154:0/1375955729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac00fe9a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.721+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1375955729 >> 172.21.15.154:0/1375955729 conn(0x7f2ac00f63c0 msgr2=0x7f2ac00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.721+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1375955729 shutdown_connections 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.721+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1375955729 wait complete. 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2ac4a82700 1 Processor -- start 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2ac4a82700 1 -- start start 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2ac4a82700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2ac4a82700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac010af00 con 0x7f2ac0070370 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2abe59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40214/0 (socket says 172.21.15.154:40214) 2024-01-16T17:10:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2abe59c700 1 -- 172.21.15.154:0/1391375206 learned_addr learned my addr 172.21.15.154:0/1391375206 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2abe59c700 1 -- 172.21.15.154:0/1391375206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab000b200 con 0x7f2ac0070370 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2abe59c700 1 --2- 172.21.15.154:0/1391375206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac010aa10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2ab000b1d0 tx=0x7f2ab0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2aaeffd700 1 -- 172.21.15.154:0/1391375206 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab0011a20 con 0x7f2ac0070370 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2aaeffd700 1 -- 172.21.15.154:0/1391375206 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2ab001b880 con 0x7f2ac0070370 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.722+0000 7f2aaeffd700 1 -- 172.21.15.154:0/1391375206 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2ab001a770 con 0x7f2ac0070370 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.723+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac010b100 con 0x7f2ac0070370 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.723+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac010b510 con 0x7f2ac0070370 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.723+0000 7f2aaeffd700 1 -- 172.21.15.154:0/1391375206 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2ab000f070 con 0x7f2ac0070370 2024-01-16T17:10:08.708 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.723+0000 7f2aaeffd700 1 -- 172.21.15.154:0/1391375206 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2ab0025a40 con 0x7f2ac0070370 2024-01-16T17:10:08.709 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.724+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac0045ad0 con 0x7f2ac0070370 2024-01-16T17:10:08.709 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.726+0000 7f2aaeffd700 1 -- 172.21.15.154:0/1391375206 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2ab0016070 con 0x7f2ac0070370 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.356+0000 7faadcba2700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.356+0000 7faadcba2700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.356+0000 7faadcba2700 0 pidfile_write: ignore empty --pid-file 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 bash[67508]: cd4f05a5008b902f6396ac61558198d5de9abeb205bddb1ef444d16e3341d951 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 systemd[1]: Started Ceph mon.smithi154 for 059f8940-b492-11ee-95ae-87774f69a715. 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.360+0000 7faadcba2700 0 load: jerasure load: lrc load: isa 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: RocksDB version: 6.8.1 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Compile date May 13 2021 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: DB SUMMARY 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: CURRENT file: CURRENT 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi154/store.db dir, Total Num: 1, files: 000004.sst 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi154/store.db: 000006.log size: 92938 ; 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.error_if_exists: 0 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.create_if_missing: 0 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.paranoid_checks: 1 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.env: 0x5565ca8fa1c0 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.fs: Posix File System 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.info_log: 0x5565cc9bd960 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.statistics: (nil) 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.use_fsync: 0 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_log_file_size: 0 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-16T17:10:08.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.allow_fallocate: 1 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.use_direct_reads: 0 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.db_log_dir: 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi154/store.db 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_subcompactions: 1 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_background_flushes: -1 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-16T17:10:08.714 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.write_buffer_manager: 0x5565cc9c3fb0 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.unordered_write: 0 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-16T17:10:08.715 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.row_cache: None 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.wal_filter: None 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.preserve_deletes: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.two_write_queues: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.atomic_flush: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.log_readahead_size: 0 2024-01-16T17:10:08.716 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_background_jobs: 2 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_background_compactions: -1 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_open_files: -1 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-16T17:10:08.717 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Compression algorithms supported: 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: kZSTD supported: 0 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: kXpressCompression supported: 0 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: kLZ4Compression supported: 1 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: kBZip2Compression supported: 0 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: kZlibCompression supported: 1 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: kSnappyCompression supported: 1 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi154/store.db/MANIFEST-000005 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.718 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.merge_operator: 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_filter: None 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5565cc8d4128) 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cache_index_and_filter_blocks: 1 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: pin_top_level_index_and_filter: 1 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: index_type: 0 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: data_block_index_type: 0 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: index_shortening: 1 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: data_block_hash_table_util_ratio: 0.750000 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: hash_index_allow_collision: 1 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: checksum: 1 2024-01-16T17:10:08.719 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: no_block_cache: 0 2024-01-16T17:10:08.720 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: block_cache: 0x5565cc90af10 2024-01-16T17:10:08.720 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: block_cache_name: BinnedLRUCache 2024-01-16T17:10:08.720 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: block_cache_options: 2024-01-16T17:10:08.720 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: capacity : 536870912 2024-01-16T17:10:08.720 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: num_shard_bits : 4 2024-01-16T17:10:08.721 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: strict_capacity_limit : 0 2024-01-16T17:10:08.721 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: high_pri_pool_ratio: 0.000 2024-01-16T17:10:08.721 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: block_cache_compressed: (nil) 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: persistent_cache: (nil) 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: block_size: 4096 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: block_size_deviation: 10 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: block_restart_interval: 16 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: index_block_restart_interval: 1 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: metadata_block_size: 4096 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: partition_filters: 0 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: use_delta_encoding: 1 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: whole_key_filtering: 1 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: verify_compression: 0 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: read_amp_bytes_per_bit: 0 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: format_version: 2 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: enable_index_compression: 1 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: block_align: 0 2024-01-16T17:10:08.722 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compression: NoCompression 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.num_levels: 7 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-16T17:10:08.723 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-16T17:10:08.724 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-16T17:10:08.725 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.table_properties_collectors: 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.inplace_update_support: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.bloom_locality: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.max_successive_merges: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-16T17:10:08.726 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.ttl: 2592000 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.361+0000 7faadcba2700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.362+0000 7faadcba2700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi154/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.362+0000 7faadcba2700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.362+0000 7faadcba2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705425008363838, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.362+0000 7faadcba2700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.363+0000 7faadcba2700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.363+0000 7faadcba2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705425008364722, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88718, "table_properties": {"data_size": 86833, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79974, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1705425008, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.363+0000 7faadcba2700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.363+0000 7faadcba2700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1705425008365121, "job": 1, "event": "recovery_finished"} 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.364+0000 7faadcba2700 4 rocksdb: DB pointer 0x5565cc9cf800 2024-01-16T17:10:08.727 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.364+0000 7faac39ba700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.364+0000 7faac39ba700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ** DB Stats ** 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ** Compaction Stats [default] ** 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.5 0.00 0.00 1 0.001 0 0 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.5 0.00 0.00 1 0.001 0 0 2024-01-16T17:10:08.728 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.5 0.00 0.00 1 0.001 0 0 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ** Compaction Stats [default] ** 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.5 0.00 0.00 1 0.001 0 0 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: AddFile(Total Files): cumulative 0, interval 0 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: AddFile(Keys): cumulative 0, interval 0 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Cumulative compaction: 0.00 GB write, 29.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Interval compaction: 0.00 GB write, 29.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ** File Read Latency Histogram By Level [default] ** 2024-01-16T17:10:08.729 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ** Compaction Stats [default] ** 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.5 0.00 0.00 1 0.001 0 0 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.5 0.00 0.00 1 0.001 0 0 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ** Compaction Stats [default] ** 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.5 0.00 0.00 1 0.001 0 0 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: AddFile(Total Files): cumulative 0, interval 0 2024-01-16T17:10:08.730 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: AddFile(Keys): cumulative 0, interval 0 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Cumulative compaction: 0.00 GB write, 28.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ** File Read Latency Histogram By Level [default] ** 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.365+0000 7faadcba2700 0 starting mon.smithi154 rank 0 at public addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] at bind addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi154 fsid 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.366+0000 7faadcba2700 1 mon.smithi154@-1(???) e1 preinit fsid 059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.366+0000 7faadcba2700 0 mon.smithi154@-1(???).mds e1 new map 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.366+0000 7faadcba2700 0 mon.smithi154@-1(???).mds e1 print_map 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: e1 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: legacy client fscid: -1 2024-01-16T17:10:08.731 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: No filesystems configured 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.366+0000 7faadcba2700 0 mon.smithi154@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.366+0000 7faadcba2700 0 mon.smithi154@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.366+0000 7faadcba2700 0 mon.smithi154@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.366+0000 7faadcba2700 0 mon.smithi154@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:06.666661+0000 mon.smithi154 (mon.0) 8 : audit [INF] from='client.? 172.21.15.154:0/2965030653' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:06.669725+0000 mon.smithi154 (mon.0) 9 : audit [INF] from='client.? 172.21.15.154:0/2965030653' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:10:08.366+0000 7faadcba2700 1 mon.smithi154@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:08.382912+0000 mon.smithi154 (mon.0) 1 : cluster [INF] mon.smithi154 is new leader, mons smithi154 in quorum (ranks 0) 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:08.382994+0000 mon.smithi154 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi154=[v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0]} 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:08.383036+0000 mon.smithi154 (mon.0) 3 : cluster [DBG] fsmap 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:08.383064+0000 mon.smithi154 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-16T17:10:08.732 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:08.383681+0000 mon.smithi154 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-16T17:10:08.781 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.799+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f2ac01048a0 con 0x7f2ac0070370 2024-01-16T17:10:08.784 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.802+0000 7f2aaeffd700 1 -- 172.21.15.154:0/1391375206 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2ab001aec0 con 0x7f2ac0070370 2024-01-16T17:10:08.785 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.802+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 msgr2=0x7f2ac010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:08.785 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.802+0000 7f2ac4a82700 1 --2- 172.21.15.154:0/1391375206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac010aa10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f2ab000b1d0 tx=0x7f2ab0006ee0).stop 2024-01-16T17:10:08.785 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.802+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 shutdown_connections 2024-01-16T17:10:08.786 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.803+0000 7f2ac4a82700 1 --2- 172.21.15.154:0/1391375206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ac0070370 0x7f2ac010aa10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:08.786 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.803+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 >> 172.21.15.154:0/1391375206 conn(0x7f2ac00f63c0 msgr2=0x7f2ac00f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:08.786 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.803+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 shutdown_connections 2024-01-16T17:10:08.786 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:08.803+0000 7f2ac4a82700 1 -- 172.21.15.154:0/1391375206 wait complete. 2024-01-16T17:10:09.150 INFO:teuthology.orchestra.run.smithi154.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-16T17:10:09.150 INFO:teuthology.orchestra.run.smithi154.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-16T17:10:09.150 INFO:teuthology.orchestra.run.smithi154.stderr:Creating mgr... 2024-01-16T17:10:09.151 INFO:teuthology.orchestra.run.smithi154.stderr:Verifying port 9283 ... 2024-01-16T17:10:09.152 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl daemon-reload 2024-01-16T17:10:09.248 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mgr.smithi154.sllivz 2024-01-16T17:10:09.270 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl reset-failed ceph-059f8940-b492-11ee-95ae-87774f69a715@mgr.smithi154.sllivz 2024-01-16T17:10:09.276 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Failed to reset failed state of unit ceph-059f8940-b492-11ee-95ae-87774f69a715@mgr.smithi154.sllivz.service: Unit ceph-059f8940-b492-11ee-95ae-87774f69a715@mgr.smithi154.sllivz.service not loaded. 2024-01-16T17:10:09.276 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl enable ceph-059f8940-b492-11ee-95ae-87774f69a715@mgr.smithi154.sllivz 2024-01-16T17:10:09.282 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: Created symlink /etc/systemd/system/ceph-059f8940-b492-11ee-95ae-87774f69a715.target.wants/ceph-059f8940-b492-11ee-95ae-87774f69a715@mgr.smithi154.sllivz.service → /etc/systemd/system/ceph-059f8940-b492-11ee-95ae-87774f69a715@.service. 2024-01-16T17:10:09.367 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl start ceph-059f8940-b492-11ee-95ae-87774f69a715@mgr.smithi154.sllivz 2024-01-16T17:10:09.685 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-16T17:10:09.694 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: disabled 2024-01-16T17:10:09.694 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-active firewalld.service 2024-01-16T17:10:09.702 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: inactive 2024-01-16T17:10:09.703 INFO:teuthology.orchestra.run.smithi154.stderr:firewalld.service is not enabled 2024-01-16T17:10:09.703 INFO:teuthology.orchestra.run.smithi154.stderr:Not possible to enable service . firewalld.service is not available 2024-01-16T17:10:09.703 INFO:teuthology.orchestra.run.smithi154.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-16T17:10:09.704 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-16T17:10:09.711 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: disabled 2024-01-16T17:10:09.712 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-active firewalld.service 2024-01-16T17:10:09.719 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: inactive 2024-01-16T17:10:09.720 INFO:teuthology.orchestra.run.smithi154.stderr:firewalld.service is not enabled 2024-01-16T17:10:09.720 INFO:teuthology.orchestra.run.smithi154.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-16T17:10:09.720 INFO:teuthology.orchestra.run.smithi154.stderr:Waiting for mgr to start... 2024-01-16T17:10:09.720 INFO:teuthology.orchestra.run.smithi154.stderr:Waiting for mgr... 2024-01-16T17:10:09.720 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-16T17:10:10.106 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.122+0000 7efc5905c700 1 Processor -- start 2024-01-16T17:10:10.106 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.122+0000 7efc5905c700 1 -- start start 2024-01-16T17:10:10.106 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.122+0000 7efc5905c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc54100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:10.107 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.122+0000 7efc5905c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc540fc710 con 0x7efc540703f0 2024-01-16T17:10:10.107 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.123+0000 7efc52d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc54100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:10.109 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.123+0000 7efc52d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc54100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40246/0 (socket says 172.21.15.154:40246) 2024-01-16T17:10:10.109 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.123+0000 7efc52d9d700 1 -- 172.21.15.154:0/1535754711 learned_addr learned my addr 172.21.15.154:0/1535754711 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:10.109 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.123+0000 7efc52d9d700 1 -- 172.21.15.154:0/1535754711 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54101210 con 0x7efc540703f0 2024-01-16T17:10:10.109 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.123+0000 7efc52d9d700 1 --2- 172.21.15.154:0/1535754711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc54100d20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7efc4400b5f0 tx=0x7efc44006ee0).ready entity=mon.0 client_cookie=bf89532d23452550 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc51d9b700 1 -- 172.21.15.154:0/1535754711 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc4400d600 con 0x7efc540703f0 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc51d9b700 1 -- 172.21.15.154:0/1535754711 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efc4400dbd0 con 0x7efc540703f0 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc51d9b700 1 -- 172.21.15.154:0/1535754711 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc44011c00 con 0x7efc540703f0 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc5905c700 1 -- 172.21.15.154:0/1535754711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 msgr2=0x7efc54100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc5905c700 1 --2- 172.21.15.154:0/1535754711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc54100d20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7efc4400b5f0 tx=0x7efc44006ee0).stop 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc5905c700 1 -- 172.21.15.154:0/1535754711 shutdown_connections 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc5905c700 1 --2- 172.21.15.154:0/1535754711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc54100d20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc5905c700 1 -- 172.21.15.154:0/1535754711 >> 172.21.15.154:0/1535754711 conn(0x7efc540f63c0 msgr2=0x7efc540f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc5905c700 1 -- 172.21.15.154:0/1535754711 shutdown_connections 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.124+0000 7efc5905c700 1 -- 172.21.15.154:0/1535754711 wait complete. 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.125+0000 7efc5905c700 1 Processor -- start 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.125+0000 7efc5905c700 1 -- start start 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.125+0000 7efc5905c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc5410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.125+0000 7efc5905c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5410d1e0 con 0x7efc540703f0 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.125+0000 7efc52d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc5410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.125+0000 7efc52d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc5410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40248/0 (socket says 172.21.15.154:40248) 2024-01-16T17:10:10.110 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.125+0000 7efc52d9d700 1 -- 172.21.15.154:0/1945698787 learned_addr learned my addr 172.21.15.154:0/1945698787 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc52d9d700 1 -- 172.21.15.154:0/1945698787 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4400b200 con 0x7efc540703f0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc52d9d700 1 --2- 172.21.15.154:0/1945698787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc5410ccf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7efc4400e040 tx=0x7efc44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc3b7fe700 1 -- 172.21.15.154:0/1945698787 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc4400d9f0 con 0x7efc540703f0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5410d3e0 con 0x7efc540703f0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc3b7fe700 1 -- 172.21.15.154:0/1945698787 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efc4401a9f0 con 0x7efc540703f0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc3b7fe700 1 -- 172.21.15.154:0/1945698787 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc4401b8c0 con 0x7efc540703f0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5410d7f0 con 0x7efc540703f0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc3b7fe700 1 -- 172.21.15.154:0/1945698787 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7efc4400f070 con 0x7efc540703f0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.126+0000 7efc3b7fe700 1 -- 172.21.15.154:0/1945698787 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7efc44025b30 con 0x7efc540703f0 2024-01-16T17:10:10.111 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.127+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc54045ad0 con 0x7efc540703f0 2024-01-16T17:10:10.113 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.131+0000 7efc3b7fe700 1 -- 172.21.15.154:0/1945698787 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7efc44016070 con 0x7efc540703f0 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.199+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7efc5410e6f0 con 0x7efc540703f0 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.199+0000 7efc3b7fe700 1 -- 172.21.15.154:0/1945698787 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7efc4401bd20 con 0x7efc540703f0 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: { 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "fsid": "059f8940-b492-11ee-95ae-87774f69a715", 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "health": { 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "checks": {}, 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "mutes": [] 2024-01-16T17:10:10.187 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum": [ 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 0 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "smithi154" 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "monmap": { 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osdmap": { 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-16T17:10:10.188 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "pgmap": { 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "fsmap": { 2024-01-16T17:10:10.189 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "available": false, 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "modules": [ 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "iostat", 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "restful" 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "services": {} 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "servicemap": { 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "modified": "2024-01-16T17:10:05.456389+0000", 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "services": {} 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:10.190 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-16T17:10:10.191 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: } 2024-01-16T17:10:10.191 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.200+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 msgr2=0x7efc5410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:10.191 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.200+0000 7efc5905c700 1 --2- 172.21.15.154:0/1945698787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc5410ccf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7efc4400e040 tx=0x7efc44006ee0).stop 2024-01-16T17:10:10.191 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.200+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 shutdown_connections 2024-01-16T17:10:10.191 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.200+0000 7efc5905c700 1 --2- 172.21.15.154:0/1945698787 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc540703f0 0x7efc5410ccf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:10.191 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.200+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 >> 172.21.15.154:0/1945698787 conn(0x7efc540f63c0 msgr2=0x7efc540f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:10.191 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.200+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 shutdown_connections 2024-01-16T17:10:10.191 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:10.200+0000 7efc5905c700 1 -- 172.21.15.154:0/1945698787 wait complete. 2024-01-16T17:10:10.209 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:08.803216+0000 mon.smithi154 (mon.0) 6 : audit [INF] from='client.? 172.21.15.154:0/1391375206' entity='client.admin' 2024-01-16T17:10:10.542 INFO:teuthology.orchestra.run.smithi154.stderr:mgr not available, waiting (1/15)... 2024-01-16T17:10:11.210 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:10 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:10.200606+0000 mon.smithi154 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.154:0/1945698787' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-16T17:10:12.544 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-16T17:10:12.908 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.923+0000 7f4892dba700 1 Processor -- start 2024-01-16T17:10:12.908 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.923+0000 7f4892dba700 1 -- start start 2024-01-16T17:10:12.908 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.923+0000 7f4892dba700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.923+0000 7f4892dba700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c0fedd0 con 0x7f488c070430 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.924+0000 7f4890b56700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.924+0000 7f4890b56700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40258/0 (socket says 172.21.15.154:40258) 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.924+0000 7f4890b56700 1 -- 172.21.15.154:0/4074015737 learned_addr learned my addr 172.21.15.154:0/4074015737 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.924+0000 7f4890b56700 1 -- 172.21.15.154:0/4074015737 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0fef10 con 0x7f488c070430 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.924+0000 7f4890b56700 1 --2- 172.21.15.154:0/4074015737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c0fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f488000b5f0 tx=0x7f4880006ee0).ready entity=mon.0 client_cookie=13fe30bb5a6a5edc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.924+0000 7f488b7fe700 1 -- 172.21.15.154:0/4074015737 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488000d600 con 0x7f488c070430 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.924+0000 7f488b7fe700 1 -- 172.21.15.154:0/4074015737 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f488000dbd0 con 0x7f488c070430 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.924+0000 7f488b7fe700 1 -- 172.21.15.154:0/4074015737 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4880011c00 con 0x7f488c070430 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.925+0000 7f4892dba700 1 -- 172.21.15.154:0/4074015737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 msgr2=0x7f488c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.925+0000 7f4892dba700 1 --2- 172.21.15.154:0/4074015737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c0fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f488000b5f0 tx=0x7f4880006ee0).stop 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.925+0000 7f4892dba700 1 -- 172.21.15.154:0/4074015737 shutdown_connections 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.925+0000 7f4892dba700 1 --2- 172.21.15.154:0/4074015737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c0fe8e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.925+0000 7f4892dba700 1 -- 172.21.15.154:0/4074015737 >> 172.21.15.154:0/4074015737 conn(0x7f488c0f63c0 msgr2=0x7f488c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.925+0000 7f4892dba700 1 -- 172.21.15.154:0/4074015737 shutdown_connections 2024-01-16T17:10:12.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.925+0000 7f4892dba700 1 -- 172.21.15.154:0/4074015737 wait complete. 2024-01-16T17:10:12.910 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4892dba700 1 Processor -- start 2024-01-16T17:10:12.910 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4892dba700 1 -- start start 2024-01-16T17:10:12.910 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4892dba700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:12.910 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4892dba700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c10aef0 con 0x7f488c070430 2024-01-16T17:10:12.910 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4890b56700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:12.910 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4890b56700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:40262/0 (socket says 172.21.15.154:40262) 2024-01-16T17:10:12.910 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4890b56700 1 -- 172.21.15.154:0/631968997 learned_addr learned my addr 172.21.15.154:0/631968997 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4890b56700 1 -- 172.21.15.154:0/631968997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488000b200 con 0x7f488c070430 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.926+0000 7f4890b56700 1 --2- 172.21.15.154:0/631968997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c10aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f488000b190 tx=0x7f4880006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.927+0000 7f48897fa700 1 -- 172.21.15.154:0/631968997 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488001b410 con 0x7f488c070430 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.927+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c10b0f0 con 0x7f488c070430 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.927+0000 7f48897fa700 1 -- 172.21.15.154:0/631968997 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f488001b9e0 con 0x7f488c070430 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.927+0000 7f48897fa700 1 -- 172.21.15.154:0/631968997 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488001a880 con 0x7f488c070430 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.927+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c10b500 con 0x7f488c070430 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.927+0000 7f48897fa700 1 -- 172.21.15.154:0/631968997 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f488000f070 con 0x7f488c070430 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.927+0000 7f48897fa700 1 -- 172.21.15.154:0/631968997 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4880025c90 con 0x7f488c070430 2024-01-16T17:10:12.911 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.928+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c045ad0 con 0x7f488c070430 2024-01-16T17:10:12.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.930+0000 7f48897fa700 1 -- 172.21.15.154:0/631968997 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4880016070 con 0x7f488c070430 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.998+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f488c104890 con 0x7f488c070430 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.999+0000 7f48897fa700 1 -- 172.21.15.154:0/631968997 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f488001bb50 con 0x7f488c070430 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: { 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "fsid": "059f8940-b492-11ee-95ae-87774f69a715", 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "health": { 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "checks": {}, 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "mutes": [] 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-16T17:10:12.982 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum": [ 2024-01-16T17:10:12.987 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 0 2024-01-16T17:10:12.987 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:12.987 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "smithi154" 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "monmap": { 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osdmap": { 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-16T17:10:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "pgmap": { 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "fsmap": { 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-16T17:10:12.989 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "available": false, 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "modules": [ 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "iostat", 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "restful" 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "services": {} 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "servicemap": { 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "modified": "2024-01-16T17:10:05.456389+0000", 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "services": {} 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: } 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.999+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 msgr2=0x7f488c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:12.999+0000 7f4892dba700 1 --2- 172.21.15.154:0/631968997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c10aa00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f488000b190 tx=0x7f4880006ee0).stop 2024-01-16T17:10:12.990 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:13.000+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 shutdown_connections 2024-01-16T17:10:12.991 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:13.000+0000 7f4892dba700 1 --2- 172.21.15.154:0/631968997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f488c070430 0x7f488c10aa00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:12.991 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:13.000+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 >> 172.21.15.154:0/631968997 conn(0x7f488c0f63c0 msgr2=0x7f488c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:12.991 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:13.000+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 shutdown_connections 2024-01-16T17:10:12.991 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:13.000+0000 7f4892dba700 1 -- 172.21.15.154:0/631968997 wait complete. 2024-01-16T17:10:13.285 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:13.000104+0000 mon.smithi154 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.154:0/631968997' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-16T17:10:13.316 INFO:teuthology.orchestra.run.smithi154.stderr:mgr not available, waiting (2/15)... 2024-01-16T17:10:14.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:14.622374+0000 mon.smithi154 (mon.0) 9 : cluster [INF] Activating manager daemon smithi154.sllivz 2024-01-16T17:10:14.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:14.625063+0000 mon.smithi154 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi154.sllivz(active, starting, since 0.00280331s) 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.626009+0000 mon.smithi154 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.626191+0000 mon.smithi154 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.626363+0000 mon.smithi154 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.626533+0000 mon.smithi154 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.626724+0000 mon.smithi154 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mgr metadata", "who": "smithi154.sllivz", "id": "smithi154.sllivz"}]: dispatch 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:14.641496+0000 mon.smithi154 (mon.0) 16 : cluster [INF] Manager daemon smithi154.sllivz is now available 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.648979+0000 mon.smithi154 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.653503+0000 mon.smithi154 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.654652+0000 mon.smithi154 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.659607+0000 mon.smithi154 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:14.961 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:14.661835+0000 mon.smithi154 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.154:0/4137575698' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:15.318 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-16T17:10:15.675 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.688+0000 7fd6935a7700 1 Processor -- start 2024-01-16T17:10:15.675 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.688+0000 7fd6935a7700 1 -- start start 2024-01-16T17:10:15.675 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.688+0000 7fd6935a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:15.675 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.688+0000 7fd6935a7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0fede0 con 0x7fd68c0704a0 2024-01-16T17:10:15.675 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.689+0000 7fd691343700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.689+0000 7fd691343700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42294/0 (socket says 172.21.15.154:42294) 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.689+0000 7fd691343700 1 -- 172.21.15.154:0/1049289696 learned_addr learned my addr 172.21.15.154:0/1049289696 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.689+0000 7fd691343700 1 -- 172.21.15.154:0/1049289696 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c0fef20 con 0x7fd68c0704a0 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.689+0000 7fd691343700 1 --2- 172.21.15.154:0/1049289696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fe8f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd67c00b5f0 tx=0x7fd67c006ee0).ready entity=mon.0 client_cookie=88188ac2e61c3c83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.689+0000 7fd67bfff700 1 -- 172.21.15.154:0/1049289696 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c00d600 con 0x7fd68c0704a0 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.689+0000 7fd67bfff700 1 -- 172.21.15.154:0/1049289696 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd67c00dbd0 con 0x7fd68c0704a0 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.689+0000 7fd67bfff700 1 -- 172.21.15.154:0/1049289696 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c011c00 con 0x7fd68c0704a0 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 -- 172.21.15.154:0/1049289696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 msgr2=0x7fd68c0fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 --2- 172.21.15.154:0/1049289696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fe8f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd67c00b5f0 tx=0x7fd67c006ee0).stop 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 -- 172.21.15.154:0/1049289696 shutdown_connections 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 --2- 172.21.15.154:0/1049289696 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fe8f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 -- 172.21.15.154:0/1049289696 >> 172.21.15.154:0/1049289696 conn(0x7fd68c0f63b0 msgr2=0x7fd68c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 -- 172.21.15.154:0/1049289696 shutdown_connections 2024-01-16T17:10:15.676 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 -- 172.21.15.154:0/1049289696 wait complete. 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 Processor -- start 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.690+0000 7fd6935a7700 1 -- start start 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd6935a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd6935a7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0fa7f0 con 0x7fd68c0704a0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd691343700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd691343700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42300/0 (socket says 172.21.15.154:42300) 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd691343700 1 -- 172.21.15.154:0/2814916906 learned_addr learned my addr 172.21.15.154:0/2814916906 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd691343700 1 -- 172.21.15.154:0/2814916906 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c00b200 con 0x7fd68c0704a0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd691343700 1 --2- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fc370 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd67c00e040 tx=0x7fd67c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd679ffb700 1 -- 172.21.15.154:0/2814916906 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c00d9f0 con 0x7fd68c0704a0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c0fa990 con 0x7fd68c0704a0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd679ffb700 1 -- 172.21.15.154:0/2814916906 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd67c01a9f0 con 0x7fd68c0704a0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd679ffb700 1 -- 172.21.15.154:0/2814916906 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c01b8c0 con 0x7fd68c0704a0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.691+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c0fada0 con 0x7fd68c0704a0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.692+0000 7fd679ffb700 1 -- 172.21.15.154:0/2814916906 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fd67c00f070 con 0x7fd68c0704a0 2024-01-16T17:10:15.677 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.692+0000 7fd679ffb700 1 --2- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fd67402fcc0 0x7fd674032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:15.678 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.692+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68c045ad0 con 0x7fd68c0704a0 2024-01-16T17:10:15.678 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.693+0000 7fd679ffb700 1 -- 172.21.15.154:0/2814916906 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd67c0410c0 con 0x7fd68c0704a0 2024-01-16T17:10:15.678 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.693+0000 7fd690b42700 1 --2- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fd67402fcc0 0x7fd674032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:15.678 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.693+0000 7fd690b42700 1 --2- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fd67402fcc0 0x7fd674032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd6880096a0 tx=0x7fd688006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:15.679 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.697+0000 7fd679ffb700 1 -- 172.21.15.154:0/2814916906 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd67c01bcb0 con 0x7fd68c0704a0 2024-01-16T17:10:15.884 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.901+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd68c0ffc80 con 0x7fd68c0704a0 2024-01-16T17:10:15.884 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.901+0000 7fd679ffb700 1 -- 172.21.15.154:0/2814916906 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7fd67c022310 con 0x7fd68c0704a0 2024-01-16T17:10:15.884 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.884 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: { 2024-01-16T17:10:15.885 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "fsid": "059f8940-b492-11ee-95ae-87774f69a715", 2024-01-16T17:10:15.885 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "health": { 2024-01-16T17:10:15.885 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-16T17:10:15.885 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "checks": {}, 2024-01-16T17:10:15.885 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "mutes": [] 2024-01-16T17:10:15.885 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:15.890 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-16T17:10:15.890 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum": [ 2024-01-16T17:10:15.890 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 0 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "smithi154" 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "quorum_age": 7, 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "monmap": { 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osdmap": { 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-16T17:10:15.891 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "pgmap": { 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "fsmap": { 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-16T17:10:15.892 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "available": true, 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "modules": [ 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "iostat", 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "restful" 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ], 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "services": {} 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "servicemap": { 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "modified": "2024-01-16T17:10:05.456389+0000", 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "services": {} 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: }, 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: } 2024-01-16T17:10:15.893 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fd67402fcc0 msgr2=0x7fd674032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 --2- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fd67402fcc0 0x7fd674032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd6880096a0 tx=0x7fd688006b40).stop 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 msgr2=0x7fd68c0fc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 --2- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fc370 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd67c00e040 tx=0x7fd67c006ee0).stop 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 shutdown_connections 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 --2- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fd67402fcc0 0x7fd674032180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 --2- 172.21.15.154:0/2814916906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd68c0704a0 0x7fd68c0fc370 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 >> 172.21.15.154:0/2814916906 conn(0x7fd68c0f63b0 msgr2=0x7fd68c0f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 shutdown_connections 2024-01-16T17:10:15.894 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:15.903+0000 7fd6935a7700 1 -- 172.21.15.154:0/2814916906 wait complete. 2024-01-16T17:10:16.224 INFO:teuthology.orchestra.run.smithi154.stderr:mgr is available 2024-01-16T17:10:16.224 INFO:teuthology.orchestra.run.smithi154.stderr:Enabling cephadm module... 2024-01-16T17:10:16.224 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-16T17:10:16.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.605+0000 7fa6c6546700 1 Processor -- start 2024-01-16T17:10:16.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.605+0000 7fa6c6546700 1 -- start start 2024-01-16T17:10:16.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.605+0000 7fa6c6546700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:16.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.605+0000 7fa6c6546700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c00fb490 con 0x7fa6c00fab80 2024-01-16T17:10:16.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.605+0000 7fa6bffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:16.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.606+0000 7fa6bffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42314/0 (socket says 172.21.15.154:42314) 2024-01-16T17:10:16.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.606+0000 7fa6bffff700 1 -- 172.21.15.154:0/2317780457 learned_addr learned my addr 172.21.15.154:0/2317780457 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:16.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.606+0000 7fa6bffff700 1 -- 172.21.15.154:0/2317780457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c00fb5d0 con 0x7fa6c00fab80 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.606+0000 7fa6bffff700 1 --2- 172.21.15.154:0/2317780457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c00fafa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa6b000b5f0 tx=0x7fa6b0006ee0).ready entity=mon.0 client_cookie=62789155a93a7077 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.606+0000 7fa6beffd700 1 -- 172.21.15.154:0/2317780457 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b000d600 con 0x7fa6c00fab80 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.606+0000 7fa6beffd700 1 -- 172.21.15.154:0/2317780457 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa6b000dbd0 con 0x7fa6c00fab80 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.606+0000 7fa6beffd700 1 -- 172.21.15.154:0/2317780457 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b0011c00 con 0x7fa6c00fab80 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 -- 172.21.15.154:0/2317780457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 msgr2=0x7fa6c00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 --2- 172.21.15.154:0/2317780457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c00fafa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa6b000b5f0 tx=0x7fa6b0006ee0).stop 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 -- 172.21.15.154:0/2317780457 shutdown_connections 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 --2- 172.21.15.154:0/2317780457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c00fafa0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 -- 172.21.15.154:0/2317780457 >> 172.21.15.154:0/2317780457 conn(0x7fa6c00f63c0 msgr2=0x7fa6c00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 -- 172.21.15.154:0/2317780457 shutdown_connections 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 -- 172.21.15.154:0/2317780457 wait complete. 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 Processor -- start 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 -- start start 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c010a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.607+0000 7fa6c6546700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c010aee0 con 0x7fa6c00fab80 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6bffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c010a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:16.591 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6bffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c010a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42322/0 (socket says 172.21.15.154:42322) 2024-01-16T17:10:16.592 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6bffff700 1 -- 172.21.15.154:0/1502377635 learned_addr learned my addr 172.21.15.154:0/1502377635 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:16.592 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6bffff700 1 -- 172.21.15.154:0/1502377635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b000b200 con 0x7fa6c00fab80 2024-01-16T17:10:16.592 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6bffff700 1 --2- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c010a9f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa6b000b1d0 tx=0x7fa6b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6bcff9700 1 -- 172.21.15.154:0/1502377635 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b001b410 con 0x7fa6c00fab80 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6bcff9700 1 -- 172.21.15.154:0/1502377635 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa6b001b9e0 con 0x7fa6c00fab80 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6bcff9700 1 -- 172.21.15.154:0/1502377635 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6b001a840 con 0x7fa6c00fab80 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c010b0e0 con 0x7fa6c00fab80 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.608+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c010b4f0 con 0x7fa6c00fab80 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.609+0000 7fa6bcff9700 1 -- 172.21.15.154:0/1502377635 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fa6b000f070 con 0x7fa6c00fab80 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.609+0000 7fa6bcff9700 1 --2- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fa6ac02fca0 0x7fa6ac032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.609+0000 7fa6bcff9700 1 -- 172.21.15.154:0/1502377635 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa6b0040fe0 con 0x7fa6c00fab80 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.609+0000 7fa6bf7fe700 1 --2- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fa6ac02fca0 0x7fa6ac032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.609+0000 7fa6bf7fe700 1 --2- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fa6ac02fca0 0x7fa6ac032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa6b40096a0 tx=0x7fa6b4006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:16.593 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.609+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c0059100 con 0x7fa6c00fab80 2024-01-16T17:10:16.596 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.614+0000 7fa6bcff9700 1 -- 172.21.15.154:0/1502377635 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6b0025460 con 0x7fa6c00fab80 2024-01-16T17:10:16.780 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:16.797+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fa6c00fc890 con 0x7fa6c00fab80 2024-01-16T17:10:16.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:16 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:15.627613+0000 mon.smithi154 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi154.sllivz(active, since 1.00535s) 2024-01-16T17:10:16.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:16 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:15.902684+0000 mon.smithi154 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.154:0/2814916906' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-16T17:10:17.608 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.626+0000 7fa6bcff9700 1 -- 172.21.15.154:0/1502377635 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa6b00257e0 con 0x7fa6c00fab80 2024-01-16T17:10:17.609 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.627+0000 7fa6bcff9700 1 -- 172.21.15.154:0/1502377635 <== mon.0 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7fa6b00109b0 con 0x7fa6c00fab80 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.630+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fa6ac02fca0 msgr2=0x7fa6ac032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.630+0000 7fa6c6546700 1 --2- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fa6ac02fca0 0x7fa6ac032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa6b40096a0 tx=0x7fa6b4006b40).stop 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.630+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 msgr2=0x7fa6c010a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.630+0000 7fa6c6546700 1 --2- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c010a9f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa6b000b1d0 tx=0x7fa6b0006ee0).stop 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.630+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 shutdown_connections 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.630+0000 7fa6c6546700 1 --2- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7fa6ac02fca0 0x7fa6ac032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.630+0000 7fa6c6546700 1 --2- 172.21.15.154:0/1502377635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6c00fab80 0x7fa6c010a9f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.631+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 >> 172.21.15.154:0/1502377635 conn(0x7fa6c00f63c0 msgr2=0x7fa6c0070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.631+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 shutdown_connections 2024-01-16T17:10:17.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:17.631+0000 7fa6c6546700 1 -- 172.21.15.154:0/1502377635 wait complete. 2024-01-16T17:10:17.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:16.799307+0000 mon.smithi154 (mon.0) 24 : audit [INF] from='client.? 172.21.15.154:0/1502377635' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-16T17:10:18.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:18 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:17.628735+0000 mon.smithi154 (mon.0) 25 : audit [INF] from='client.? 172.21.15.154:0/1502377635' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-16T17:10:18.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:18 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:17.628794+0000 mon.smithi154 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi154.sllivz(active, since 3s) 2024-01-16T17:10:19.219 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.592+0000 7f3da1b78700 1 Processor -- start 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.592+0000 7f3da1b78700 1 -- start start 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3da1b78700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3da1b78700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c0fedd0 con 0x7f3d9c070430 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3d9b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3d9b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42350/0 (socket says 172.21.15.154:42350) 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3d9b7fe700 1 -- 172.21.15.154:0/1357885518 learned_addr learned my addr 172.21.15.154:0/1357885518 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3d9b7fe700 1 -- 172.21.15.154:0/1357885518 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0fef10 con 0x7f3d9c070430 2024-01-16T17:10:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3d9b7fe700 1 --2- 172.21.15.154:0/1357885518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c0fe8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3d8c00b5f0 tx=0x7f3d8c006ee0).ready entity=mon.0 client_cookie=e0dda80d6d14f38c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3d9a7fc700 1 -- 172.21.15.154:0/1357885518 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8c00d600 con 0x7f3d9c070430 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3d9a7fc700 1 -- 172.21.15.154:0/1357885518 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3d8c00dbd0 con 0x7f3d9c070430 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.593+0000 7f3d9a7fc700 1 -- 172.21.15.154:0/1357885518 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8c011c00 con 0x7f3d9c070430 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.594+0000 7f3da1b78700 1 -- 172.21.15.154:0/1357885518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 msgr2=0x7f3d9c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.594+0000 7f3da1b78700 1 --2- 172.21.15.154:0/1357885518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c0fe8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f3d8c00b5f0 tx=0x7f3d8c006ee0).stop 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.594+0000 7f3da1b78700 1 -- 172.21.15.154:0/1357885518 shutdown_connections 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.594+0000 7f3da1b78700 1 --2- 172.21.15.154:0/1357885518 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c0fe8e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.594+0000 7f3da1b78700 1 -- 172.21.15.154:0/1357885518 >> 172.21.15.154:0/1357885518 conn(0x7f3d9c0f63c0 msgr2=0x7f3d9c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.594+0000 7f3da1b78700 1 -- 172.21.15.154:0/1357885518 shutdown_connections 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.594+0000 7f3da1b78700 1 -- 172.21.15.154:0/1357885518 wait complete. 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3da1b78700 1 Processor -- start 2024-01-16T17:10:20.578 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3da1b78700 1 -- start start 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3da1b78700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3da1b78700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c10af30 con 0x7f3d9c070430 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3d9b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3d9b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42360/0 (socket says 172.21.15.154:42360) 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3d9b7fe700 1 -- 172.21.15.154:0/3814588890 learned_addr learned my addr 172.21.15.154:0/3814588890 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3d9b7fe700 1 -- 172.21.15.154:0/3814588890 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c00b200 con 0x7f3d9c070430 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.595+0000 7f3d9b7fe700 1 --2- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c10aa40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3d8c00e040 tx=0x7f3d8c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.596+0000 7f3d83fff700 1 -- 172.21.15.154:0/3814588890 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8c00d9f0 con 0x7f3d9c070430 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.596+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c10b130 con 0x7f3d9c070430 2024-01-16T17:10:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.596+0000 7f3d83fff700 1 -- 172.21.15.154:0/3814588890 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3d8c01a9f0 con 0x7f3d9c070430 2024-01-16T17:10:20.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.596+0000 7f3d83fff700 1 -- 172.21.15.154:0/3814588890 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8c01b8c0 con 0x7f3d9c070430 2024-01-16T17:10:20.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.596+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c10b540 con 0x7f3d9c070430 2024-01-16T17:10:20.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.597+0000 7f3d83fff700 1 -- 172.21.15.154:0/3814588890 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3d8c00f070 con 0x7f3d9c070430 2024-01-16T17:10:20.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.597+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7c004fa0 con 0x7f3d9c070430 2024-01-16T17:10:20.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.597+0000 7f3d83fff700 1 --2- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f3d8802fd50 0x7f3d88032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:20.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.597+0000 7f3d83fff700 1 -- 172.21.15.154:0/3814588890 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3d8c041150 con 0x7f3d9c070430 2024-01-16T17:10:20.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.597+0000 7f3d9affd700 1 -- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f3d8802fd50 msgr2=0x7f3d88032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/946610216 2024-01-16T17:10:20.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.597+0000 7f3d9affd700 1 --2- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f3d8802fd50 0x7f3d88032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-16T17:10:20.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.601+0000 7f3d83fff700 1 -- 172.21.15.154:0/3814588890 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d8c022d20 con 0x7f3d9c070430 2024-01-16T17:10:20.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.794+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3d7c005e80 con 0x7f3d9c070430 2024-01-16T17:10:20.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.794+0000 7f3d83fff700 1 -- 172.21.15.154:0/3814588890 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3d8c016070 con 0x7f3d9c070430 2024-01-16T17:10:20.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: { 2024-01-16T17:10:20.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 4, 2024-01-16T17:10:20.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "available": true, 2024-01-16T17:10:20.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "active_name": "smithi154.sllivz", 2024-01-16T17:10:20.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-16T17:10:20.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: } 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.796+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f3d8802fd50 msgr2=0x7f3d88032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.796+0000 7f3da1b78700 1 --2- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f3d8802fd50 0x7f3d88032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.796+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 msgr2=0x7f3d9c10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.796+0000 7f3da1b78700 1 --2- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c10aa40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f3d8c00e040 tx=0x7f3d8c006ee0).stop 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.796+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 shutdown_connections 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.796+0000 7f3da1b78700 1 --2- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f3d8802fd50 0x7f3d88032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.796+0000 7f3da1b78700 1 --2- 172.21.15.154:0/3814588890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d9c070430 0x7f3d9c10aa40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.796+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 >> 172.21.15.154:0/3814588890 conn(0x7f3d9c0f63c0 msgr2=0x7f3d9c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.797+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 shutdown_connections 2024-01-16T17:10:20.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:20.797+0000 7f3da1b78700 1 -- 172.21.15.154:0/3814588890 wait complete. 2024-01-16T17:10:21.067 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:20 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:20.795661+0000 mon.smithi154 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.154:0/3814588890' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-16T17:10:21.131 INFO:teuthology.orchestra.run.smithi154.stderr:Waiting for the mgr to restart... 2024-01-16T17:10:21.131 INFO:teuthology.orchestra.run.smithi154.stderr:Waiting for mgr epoch 4... 2024-01-16T17:10:21.131 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.527+0000 7f0c1501e700 1 Processor -- start 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.527+0000 7f0c1501e700 1 -- start start 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.527+0000 7f0c1501e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c10100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.527+0000 7f0c1501e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c100fc750 con 0x7f0c100703b0 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.528+0000 7f0c0ed9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c10100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.528+0000 7f0c0ed9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c10100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42376/0 (socket says 172.21.15.154:42376) 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.528+0000 7f0c0ed9d700 1 -- 172.21.15.154:0/1748850503 learned_addr learned my addr 172.21.15.154:0/1748850503 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.528+0000 7f0c0ed9d700 1 -- 172.21.15.154:0/1748850503 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c10101250 con 0x7f0c100703b0 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.528+0000 7f0c0ed9d700 1 --2- 172.21.15.154:0/1748850503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c10100d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0c0000b5f0 tx=0x7f0c00006ee0).ready entity=mon.0 client_cookie=cb59001a52caf83c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.528+0000 7f0c0dd9b700 1 -- 172.21.15.154:0/1748850503 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0000d600 con 0x7f0c100703b0 2024-01-16T17:10:21.515 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.528+0000 7f0c0dd9b700 1 -- 172.21.15.154:0/1748850503 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0c0000dbd0 con 0x7f0c100703b0 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.528+0000 7f0c0dd9b700 1 -- 172.21.15.154:0/1748850503 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c00011c00 con 0x7f0c100703b0 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.529+0000 7f0c1501e700 1 -- 172.21.15.154:0/1748850503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 msgr2=0x7f0c10100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.529+0000 7f0c1501e700 1 --2- 172.21.15.154:0/1748850503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c10100d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f0c0000b5f0 tx=0x7f0c00006ee0).stop 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.529+0000 7f0c1501e700 1 -- 172.21.15.154:0/1748850503 shutdown_connections 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.529+0000 7f0c1501e700 1 --2- 172.21.15.154:0/1748850503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c10100d60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.529+0000 7f0c1501e700 1 -- 172.21.15.154:0/1748850503 >> 172.21.15.154:0/1748850503 conn(0x7f0c100f63c0 msgr2=0x7f0c100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.529+0000 7f0c1501e700 1 -- 172.21.15.154:0/1748850503 shutdown_connections 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.529+0000 7f0c1501e700 1 -- 172.21.15.154:0/1748850503 wait complete. 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c1501e700 1 Processor -- start 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c1501e700 1 -- start start 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c1501e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c1010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c1501e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1010d100 con 0x7f0c100703b0 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c0ed9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c1010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:21.516 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c0ed9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c1010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:42378/0 (socket says 172.21.15.154:42378) 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c0ed9d700 1 -- 172.21.15.154:0/2698590214 learned_addr learned my addr 172.21.15.154:0/2698590214 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c0ed9d700 1 -- 172.21.15.154:0/2698590214 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0000b200 con 0x7f0c100703b0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c0ed9d700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c1010cc10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0c0000b190 tx=0x7f0c00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0001b410 con 0x7f0c100703b0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0c0001b9e0 con 0x7f0c100703b0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1010d300 con 0x7f0c100703b0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c0001a880 con 0x7f0c100703b0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.530+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1010d710 con 0x7f0c100703b0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.531+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0c0000f070 con 0x7f0c100703b0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.531+0000 7f0bf77fe700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 0x7f0bfc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.531+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 --> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c10059100 con 0x7f0bfc02fd50 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.531+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0c00041d70 con 0x7f0c100703b0 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.531+0000 7f0c0e59c700 1 -- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 msgr2=0x7f0bfc032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/946610216 2024-01-16T17:10:21.517 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.531+0000 7f0c0e59c700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 0x7f0bfc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-16T17:10:21.714 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.732+0000 7f0c0e59c700 1 -- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 msgr2=0x7f0bfc032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/946610216 2024-01-16T17:10:21.714 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:21.732+0000 7f0c0e59c700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 0x7f0bfc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-16T17:10:22.115 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:22.133+0000 7f0c0e59c700 1 -- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 msgr2=0x7f0bfc032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/946610216 2024-01-16T17:10:22.115 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:22.133+0000 7f0c0e59c700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 0x7f0bfc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-16T17:10:22.622 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:22.639+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f0c00025980 con 0x7f0c100703b0 2024-01-16T17:10:22.622 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:22.639+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 msgr2=0x7f0bfc032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T17:10:22.622 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:22.639+0000 7f0bf77fe700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 0x7f0bfc032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:23.626 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.643+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f0c00022310 con 0x7f0c100703b0 2024-01-16T17:10:23.626 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.643+0000 7f0bf77fe700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f0bfc033670 0x7f0bfc035a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:23.626 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.643+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0c10059100 con 0x7f0bfc033670 2024-01-16T17:10:23.626 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.643+0000 7f0c0e59c700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f0bfc033670 0x7f0bfc035a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:23.626 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.644+0000 7f0c0e59c700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f0bfc033670 0x7f0bfc035a60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0c040036d0 tx=0x7f0c04006c50).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:23.626 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.644+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f0c10059100 con 0x7f0bfc033670 2024-01-16T17:10:23.633 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.650+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0c10045ad0 con 0x7f0bfc033670 2024-01-16T17:10:23.633 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0bf77fe700 1 -- 172.21.15.154:0/2698590214 <== mgr.14114 v2:172.21.15.154:6800/3611230081 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0c10045ad0 con 0x7f0bfc033670 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: { 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "initialized": true 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: } 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f0bfc033670 msgr2=0x7f0bfc035a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f0bfc033670 0x7f0bfc035a60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f0c040036d0 tx=0x7f0c04006c50).stop 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 msgr2=0x7f0c1010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c1010cc10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0c0000b190 tx=0x7f0c00006ee0).stop 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 shutdown_connections 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f0bfc033670 0x7f0bfc035a60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c100703b0 0x7f0c1010cc10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 --2- 172.21.15.154:0/2698590214 >> [v2:172.21.15.154:6800/946610216,v1:172.21.15.154:6801/946610216] conn(0x7f0bfc02fd50 0x7f0bfc032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:23.635 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 >> 172.21.15.154:0/2698590214 conn(0x7f0c100f63c0 msgr2=0x7f0c100f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:23.636 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 shutdown_connections 2024-01-16T17:10:23.636 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:23.651+0000 7f0c1501e700 1 -- 172.21.15.154:0/2698590214 wait complete. 2024-01-16T17:10:23.925 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:22.589010+0000 mon.smithi154 (mon.0) 28 : cluster [INF] Active manager daemon smithi154.sllivz restarted 2024-01-16T17:10:23.925 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:22.589408+0000 mon.smithi154 (mon.0) 29 : cluster [INF] Activating manager daemon smithi154.sllivz 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:22.591065+0000 mon.smithi154 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:22.641665+0000 mon.smithi154 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi154.sllivz(active, starting, since 0.0523497s) 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.642485+0000 mon.smithi154 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.643070+0000 mon.smithi154 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mgr metadata", "who": "smithi154.sllivz", "id": "smithi154.sllivz"}]: dispatch 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.643717+0000 mon.smithi154 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.643865+0000 mon.smithi154 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.643994+0000 mon.smithi154 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:22.657571+0000 mon.smithi154 (mon.0) 37 : cluster [INF] Manager daemon smithi154.sllivz is now available 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.673010+0000 mon.smithi154 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.678089+0000 mon.smithi154 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.683045+0000 mon.smithi154 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.772092+0000 mon.smithi154 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:23.926 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.772774+0000 mon.smithi154 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:23.927 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.774542+0000 mon.smithi154 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:23.927 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.775919+0000 mon.smithi154 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:23.927 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.785445+0000 mon.smithi154 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:10:23.927 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:22.786559+0000 mon.smithi154 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:10:23.973 INFO:teuthology.orchestra.run.smithi154.stderr:mgr epoch 4 is available 2024-01-16T17:10:23.973 INFO:teuthology.orchestra.run.smithi154.stderr:Setting orchestrator backend to cephadm... 2024-01-16T17:10:23.973 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.380+0000 7f76f7112700 1 Processor -- start 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.380+0000 7f76f7112700 1 -- start start 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.380+0000 7f76f7112700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f00fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.380+0000 7f76f7112700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f00fedd0 con 0x7f76f0070430 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.381+0000 7f76f4eae700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f00fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.381+0000 7f76f4eae700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f00fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33790/0 (socket says 172.21.15.154:33790) 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.381+0000 7f76f4eae700 1 -- 172.21.15.154:0/3528697146 learned_addr learned my addr 172.21.15.154:0/3528697146 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.381+0000 7f76f4eae700 1 -- 172.21.15.154:0/3528697146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f00fef10 con 0x7f76f0070430 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.381+0000 7f76f4eae700 1 --2- 172.21.15.154:0/3528697146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f00fe8e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f76e400b5f0 tx=0x7f76e4006ee0).ready entity=mon.0 client_cookie=9137d2fb8b1c0fb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.381+0000 7f76ef7fe700 1 -- 172.21.15.154:0/3528697146 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e400d600 con 0x7f76f0070430 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.381+0000 7f76ef7fe700 1 -- 172.21.15.154:0/3528697146 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f76e400dbd0 con 0x7f76f0070430 2024-01-16T17:10:24.367 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.381+0000 7f76ef7fe700 1 -- 172.21.15.154:0/3528697146 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e4011c00 con 0x7f76f0070430 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 -- 172.21.15.154:0/3528697146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 msgr2=0x7f76f00fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 --2- 172.21.15.154:0/3528697146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f00fe8e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f76e400b5f0 tx=0x7f76e4006ee0).stop 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 -- 172.21.15.154:0/3528697146 shutdown_connections 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 --2- 172.21.15.154:0/3528697146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f00fe8e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 -- 172.21.15.154:0/3528697146 >> 172.21.15.154:0/3528697146 conn(0x7f76f00f63c0 msgr2=0x7f76f00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 -- 172.21.15.154:0/3528697146 shutdown_connections 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 -- 172.21.15.154:0/3528697146 wait complete. 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 Processor -- start 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.382+0000 7f76f7112700 1 -- start start 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f7112700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f7112700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f010aef0 con 0x7f76f0070430 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f4eae700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f4eae700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f010aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33792/0 (socket says 172.21.15.154:33792) 2024-01-16T17:10:24.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f4eae700 1 -- 172.21.15.154:0/4004542346 learned_addr learned my addr 172.21.15.154:0/4004542346 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f4eae700 1 -- 172.21.15.154:0/4004542346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e400b200 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f4eae700 1 --2- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f010aa00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f76e400b190 tx=0x7f76e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76ed7fa700 1 -- 172.21.15.154:0/4004542346 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e401b410 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f010b0f0 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76ed7fa700 1 -- 172.21.15.154:0/4004542346 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f76e401b9e0 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76ed7fa700 1 -- 172.21.15.154:0/4004542346 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e401a880 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.383+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f010b500 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.384+0000 7f76ed7fa700 1 -- 172.21.15.154:0/4004542346 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f76e400f070 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.384+0000 7f76ed7fa700 1 --2- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f76d402fca0 0x7f76d4032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.384+0000 7f76ed7fa700 1 -- 172.21.15.154:0/4004542346 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f76e4042050 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.384+0000 7f76effff700 1 --2- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f76d402fca0 0x7f76d4032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.384+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f0059100 con 0x7f76f0070430 2024-01-16T17:10:24.369 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.385+0000 7f76effff700 1 --2- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f76d402fca0 0x7f76d4032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f76dc0096a0 tx=0x7f76dc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:24.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.389+0000 7f76ed7fa700 1 -- 172.21.15.154:0/4004542346 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76e4010360 con 0x7f76f0070430 2024-01-16T17:10:24.546 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.563+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f76f01048f0 con 0x7f76d402fca0 2024-01-16T17:10:24.551 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.568+0000 7f76ed7fa700 1 -- 172.21.15.154:0/4004542346 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f76f01048f0 con 0x7f76d402fca0 2024-01-16T17:10:24.553 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.570+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f76d402fca0 msgr2=0x7f76d4032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.570+0000 7f76f7112700 1 --2- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f76d402fca0 0x7f76d4032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f76dc0096a0 tx=0x7f76dc006b40).stop 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.570+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 msgr2=0x7f76f010aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.570+0000 7f76f7112700 1 --2- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f010aa00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f76e400b190 tx=0x7f76e4006ee0).stop 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.571+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 shutdown_connections 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.571+0000 7f76f7112700 1 --2- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f76d402fca0 0x7f76d4032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.571+0000 7f76f7112700 1 --2- 172.21.15.154:0/4004542346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76f0070430 0x7f76f010aa00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.571+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 >> 172.21.15.154:0/4004542346 conn(0x7f76f00f63c0 msgr2=0x7f76f00f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.571+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 shutdown_connections 2024-01-16T17:10:24.554 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:24.571+0000 7f76f7112700 1 -- 172.21.15.154:0/4004542346 wait complete. 2024-01-16T17:10:24.897 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-16T17:10:24.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:24 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:23.645645+0000 mgr.smithi154.sllivz (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-16T17:10:24.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:24 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:23.646489+0000 mon.smithi154 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi154.sllivz(active, since 1.05717s) 2024-01-16T17:10:24.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:24 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:23.652362+0000 mgr.smithi154.sllivz (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-16T17:10:24.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:24 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:24.569700+0000 mon.smithi154 (mon.0) 48 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:24.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:24 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:24.573051+0000 mon.smithi154 (mon.0) 49 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.273+0000 7fd2f4d9f700 1 Processor -- start 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.273+0000 7fd2f4d9f700 1 -- start start 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.273+0000 7fd2f4d9f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fe790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.273+0000 7fd2f4d9f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00fec80 con 0x7fd2f00703d0 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.274+0000 7fd2ef7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fe790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.274+0000 7fd2ef7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fe790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33800/0 (socket says 172.21.15.154:33800) 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.274+0000 7fd2ef7fe700 1 -- 172.21.15.154:0/623872792 learned_addr learned my addr 172.21.15.154:0/623872792 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.274+0000 7fd2ef7fe700 1 -- 172.21.15.154:0/623872792 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f00fedc0 con 0x7fd2f00703d0 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.274+0000 7fd2ef7fe700 1 --2- 172.21.15.154:0/623872792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fe790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fd2d800b5f0 tx=0x7fd2d8006ee0).ready entity=mon.0 client_cookie=85006da02a3ae438 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:25.257 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.274+0000 7fd2ee7fc700 1 -- 172.21.15.154:0/623872792 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2d800d600 con 0x7fd2f00703d0 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.274+0000 7fd2ee7fc700 1 -- 172.21.15.154:0/623872792 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd2d800dbd0 con 0x7fd2f00703d0 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.274+0000 7fd2ee7fc700 1 -- 172.21.15.154:0/623872792 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2d8011c00 con 0x7fd2f00703d0 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.275+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/623872792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 msgr2=0x7fd2f00fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.275+0000 7fd2f4d9f700 1 --2- 172.21.15.154:0/623872792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fe790 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fd2d800b5f0 tx=0x7fd2d8006ee0).stop 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.275+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/623872792 shutdown_connections 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.275+0000 7fd2f4d9f700 1 --2- 172.21.15.154:0/623872792 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fe790 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.275+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/623872792 >> 172.21.15.154:0/623872792 conn(0x7fd2f00f6210 msgr2=0x7fd2f00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.275+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/623872792 shutdown_connections 2024-01-16T17:10:25.259 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.275+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/623872792 wait complete. 2024-01-16T17:10:25.260 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.276+0000 7fd2f4d9f700 1 Processor -- start 2024-01-16T17:10:25.260 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.276+0000 7fd2f4d9f700 1 -- start start 2024-01-16T17:10:25.260 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.276+0000 7fd2f4d9f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:25.260 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.276+0000 7fd2f4d9f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f00fa690 con 0x7fd2f00703d0 2024-01-16T17:10:25.260 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.276+0000 7fd2ef7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:25.260 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.276+0000 7fd2ef7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33808/0 (socket says 172.21.15.154:33808) 2024-01-16T17:10:25.260 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.276+0000 7fd2ef7fe700 1 -- 172.21.15.154:0/970279774 learned_addr learned my addr 172.21.15.154:0/970279774 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:25.260 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.277+0000 7fd2ef7fe700 1 -- 172.21.15.154:0/970279774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d800b200 con 0x7fd2f00703d0 2024-01-16T17:10:25.261 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.277+0000 7fd2ef7fe700 1 --2- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fc210 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fd2d800b190 tx=0x7fd2d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:25.261 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.277+0000 7fd2e7fff700 1 -- 172.21.15.154:0/970279774 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2d801b410 con 0x7fd2f00703d0 2024-01-16T17:10:25.261 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.277+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f00fa830 con 0x7fd2f00703d0 2024-01-16T17:10:25.261 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.277+0000 7fd2e7fff700 1 -- 172.21.15.154:0/970279774 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd2d801b9e0 con 0x7fd2f00703d0 2024-01-16T17:10:25.261 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.277+0000 7fd2e7fff700 1 -- 172.21.15.154:0/970279774 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2d801a880 con 0x7fd2f00703d0 2024-01-16T17:10:25.262 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.277+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f00fac40 con 0x7fd2f00703d0 2024-01-16T17:10:25.262 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.277+0000 7fd2e7fff700 1 -- 172.21.15.154:0/970279774 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fd2d800f070 con 0x7fd2f00703d0 2024-01-16T17:10:25.262 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.278+0000 7fd2e7fff700 1 --2- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fd2d002fca0 0x7fd2d0032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:25.262 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.278+0000 7fd2e7fff700 1 -- 172.21.15.154:0/970279774 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd2d8042050 con 0x7fd2f00703d0 2024-01-16T17:10:25.262 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.278+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f0045ad0 con 0x7fd2f00703d0 2024-01-16T17:10:25.262 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.278+0000 7fd2eeffd700 1 --2- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fd2d002fca0 0x7fd2d0032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:25.262 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.278+0000 7fd2eeffd700 1 --2- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fd2d002fca0 0x7fd2d0032160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd2e00096a0 tx=0x7fd2e0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:25.264 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.282+0000 7fd2e7fff700 1 -- 172.21.15.154:0/970279774 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2d8010360 con 0x7fd2f00703d0 2024-01-16T17:10:25.416 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.434+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2f01046b0 con 0x7fd2d002fca0 2024-01-16T17:10:25.417 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.435+0000 7fd2e7fff700 1 -- 172.21.15.154:0/970279774 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fd2f01046b0 con 0x7fd2d002fca0 2024-01-16T17:10:25.417 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: value unchanged 2024-01-16T17:10:25.419 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fd2d002fca0 msgr2=0x7fd2d0032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:25.419 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 --2- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fd2d002fca0 0x7fd2d0032160 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fd2e00096a0 tx=0x7fd2e0006b40).stop 2024-01-16T17:10:25.419 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 msgr2=0x7fd2f00fc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:25.420 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 --2- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fc210 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fd2d800b190 tx=0x7fd2d8006ee0).stop 2024-01-16T17:10:25.420 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 shutdown_connections 2024-01-16T17:10:25.420 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 --2- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fd2d002fca0 0x7fd2d0032160 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:25.420 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 --2- 172.21.15.154:0/970279774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2f00703d0 0x7fd2f00fc210 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:25.420 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 >> 172.21.15.154:0/970279774 conn(0x7fd2f00f6210 msgr2=0x7fd2f00f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:25.420 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 shutdown_connections 2024-01-16T17:10:25.420 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:25.436+0000 7fd2f4d9f700 1 -- 172.21.15.154:0/970279774 wait complete. 2024-01-16T17:10:25.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:24.566054+0000 mgr.smithi154.sllivz (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:25.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:25.570590+0000 mon.smithi154 (mon.0) 50 : cluster [DBG] mgrmap e7: smithi154.sllivz(active, since 2s) 2024-01-16T17:10:26.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:25.436106+0000 mgr.smithi154.sllivz (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:27.478 INFO:teuthology.orchestra.run.smithi154.stderr:Generating ssh key... 2024-01-16T17:10:27.478 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-16T17:10:29.012 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.937+0000 7fade247a700 1 Processor -- start 2024-01-16T17:10:29.012 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7fade247a700 1 -- start start 2024-01-16T17:10:29.012 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7fade247a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:29.012 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7fade247a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0fee10 con 0x7faddc0703f0 2024-01-16T17:10:29.013 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7faddbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:29.013 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7faddbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33824/0 (socket says 172.21.15.154:33824) 2024-01-16T17:10:29.013 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7faddbfff700 1 -- 172.21.15.154:0/3641274036 learned_addr learned my addr 172.21.15.154:0/3641274036 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:29.013 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7faddbfff700 1 -- 172.21.15.154:0/3641274036 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faddc0fef50 con 0x7faddc0703f0 2024-01-16T17:10:29.013 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7faddbfff700 1 --2- 172.21.15.154:0/3641274036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fe920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fadcc00b5f0 tx=0x7fadcc006ee0).ready entity=mon.0 client_cookie=8f90a534845bfd7b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:29.015 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.938+0000 7faddaffd700 1 -- 172.21.15.154:0/3641274036 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadcc00d600 con 0x7faddc0703f0 2024-01-16T17:10:29.015 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7faddaffd700 1 -- 172.21.15.154:0/3641274036 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fadcc00dbd0 con 0x7faddc0703f0 2024-01-16T17:10:29.015 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7faddaffd700 1 -- 172.21.15.154:0/3641274036 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadcc011c00 con 0x7faddc0703f0 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7fade247a700 1 -- 172.21.15.154:0/3641274036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 msgr2=0x7faddc0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7fade247a700 1 --2- 172.21.15.154:0/3641274036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fe920 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fadcc00b5f0 tx=0x7fadcc006ee0).stop 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7fade247a700 1 -- 172.21.15.154:0/3641274036 shutdown_connections 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7fade247a700 1 --2- 172.21.15.154:0/3641274036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fe920 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7fade247a700 1 -- 172.21.15.154:0/3641274036 >> 172.21.15.154:0/3641274036 conn(0x7faddc0f63c0 msgr2=0x7faddc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7fade247a700 1 -- 172.21.15.154:0/3641274036 shutdown_connections 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.939+0000 7fade247a700 1 -- 172.21.15.154:0/3641274036 wait complete. 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.940+0000 7fade247a700 1 Processor -- start 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.940+0000 7fade247a700 1 -- start start 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.940+0000 7fade247a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.940+0000 7fade247a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faddc0fa820 con 0x7faddc0703f0 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.940+0000 7faddbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.940+0000 7faddbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33838/0 (socket says 172.21.15.154:33838) 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.940+0000 7faddbfff700 1 -- 172.21.15.154:0/3947309432 learned_addr learned my addr 172.21.15.154:0/3947309432 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.941+0000 7faddbfff700 1 -- 172.21.15.154:0/3947309432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadcc00b200 con 0x7faddc0703f0 2024-01-16T17:10:29.016 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.941+0000 7faddbfff700 1 --2- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fc3a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fadcc00b190 tx=0x7fadcc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:29.017 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.941+0000 7fadd8ff9700 1 -- 172.21.15.154:0/3947309432 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadcc01b410 con 0x7faddc0703f0 2024-01-16T17:10:29.017 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.941+0000 7fadd8ff9700 1 -- 172.21.15.154:0/3947309432 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fadcc01b9e0 con 0x7faddc0703f0 2024-01-16T17:10:29.017 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.941+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faddc0fa9c0 con 0x7faddc0703f0 2024-01-16T17:10:29.017 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.941+0000 7fadd8ff9700 1 -- 172.21.15.154:0/3947309432 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadcc01a880 con 0x7faddc0703f0 2024-01-16T17:10:29.017 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.941+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faddc0fadd0 con 0x7faddc0703f0 2024-01-16T17:10:29.017 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.942+0000 7fadd8ff9700 1 -- 172.21.15.154:0/3947309432 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fadcc00f070 con 0x7faddc0703f0 2024-01-16T17:10:29.017 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.942+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faddc045ad0 con 0x7faddc0703f0 2024-01-16T17:10:29.018 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.942+0000 7fadd8ff9700 1 --2- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fadc802fdb0 0x7fadc8032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:29.018 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.942+0000 7fadd8ff9700 1 -- 172.21.15.154:0/3947309432 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fadcc0412a0 con 0x7faddc0703f0 2024-01-16T17:10:29.018 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.942+0000 7faddb7fe700 1 --2- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fadc802fdb0 0x7fadc8032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:29.018 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.942+0000 7faddb7fe700 1 --2- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fadc802fdb0 0x7fadc8032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fadd00096a0 tx=0x7fadd0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:29.018 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:28.946+0000 7fadd8ff9700 1 -- 172.21.15.154:0/3947309432 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadcc0224f0 con 0x7faddc0703f0 2024-01-16T17:10:29.084 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.102+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7faddc0ffd50 con 0x7fadc802fdb0 2024-01-16T17:10:29.743 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.761+0000 7fadd8ff9700 1 -- 172.21.15.154:0/3947309432 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faddc0ffd50 con 0x7fadc802fdb0 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fadc802fdb0 msgr2=0x7fadc8032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 --2- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fadc802fdb0 0x7fadc8032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fadd00096a0 tx=0x7fadd0006b40).stop 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 msgr2=0x7faddc0fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 --2- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fc3a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fadcc00b190 tx=0x7fadcc006ee0).stop 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 shutdown_connections 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 --2- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fadc802fdb0 0x7fadc8032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 --2- 172.21.15.154:0/3947309432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faddc0703f0 0x7faddc0fc3a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 >> 172.21.15.154:0/3947309432 conn(0x7faddc0f63c0 msgr2=0x7faddc0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 shutdown_connections 2024-01-16T17:10:29.746 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:29.763+0000 7fade247a700 1 -- 172.21.15.154:0/3947309432 wait complete. 2024-01-16T17:10:29.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:29.104370+0000 mgr.smithi154.sllivz (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-16T17:10:31.210 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:30 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:29.760132+0000 mon.smithi154 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:31.210 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:30 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:29.761898+0000 mon.smithi154 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:31.580 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.692+0000 7f8b8b591700 1 Processor -- start 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.692+0000 7f8b8b591700 1 -- start start 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.692+0000 7f8b8b591700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b84070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.692+0000 7f8b8b591700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b840728d0 con 0x7f8b840724b0 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.693+0000 7f8b8932d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b84070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.693+0000 7f8b8932d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b84070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33848/0 (socket says 172.21.15.154:33848) 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.693+0000 7f8b8932d700 1 -- 172.21.15.154:0/265659597 learned_addr learned my addr 172.21.15.154:0/265659597 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.693+0000 7f8b8932d700 1 -- 172.21.15.154:0/265659597 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84070f40 con 0x7f8b840724b0 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.693+0000 7f8b8932d700 1 --2- 172.21.15.154:0/265659597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b84070a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8b8000b5f0 tx=0x7f8b80006ee0).ready entity=mon.0 client_cookie=35ac4cc12f15eeca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.693+0000 7f8b77fff700 1 -- 172.21.15.154:0/265659597 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8000d600 con 0x7f8b840724b0 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.693+0000 7f8b77fff700 1 -- 172.21.15.154:0/265659597 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8b8000dbd0 con 0x7f8b840724b0 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.693+0000 7f8b77fff700 1 -- 172.21.15.154:0/265659597 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b80011c00 con 0x7f8b840724b0 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 -- 172.21.15.154:0/265659597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 msgr2=0x7f8b84070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:32.857 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 --2- 172.21.15.154:0/265659597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b84070a50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8b8000b5f0 tx=0x7f8b80006ee0).stop 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 -- 172.21.15.154:0/265659597 shutdown_connections 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 --2- 172.21.15.154:0/265659597 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b84070a50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 -- 172.21.15.154:0/265659597 >> 172.21.15.154:0/265659597 conn(0x7f8b840f74c0 msgr2=0x7f8b840f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 -- 172.21.15.154:0/265659597 shutdown_connections 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 -- 172.21.15.154:0/265659597 wait complete. 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 Processor -- start 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 -- start start 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b8410ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.694+0000 7f8b8b591700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8410d1c0 con 0x7f8b840724b0 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b8932d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b8410ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b8932d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b8410ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:33860/0 (socket says 172.21.15.154:33860) 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b8932d700 1 -- 172.21.15.154:0/3499930979 learned_addr learned my addr 172.21.15.154:0/3499930979 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b8932d700 1 -- 172.21.15.154:0/3499930979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8000b200 con 0x7f8b840724b0 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b8932d700 1 --2- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b8410ccd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f8b8000b190 tx=0x7f8b80006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b75ffb700 1 -- 172.21.15.154:0/3499930979 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8001b410 con 0x7f8b840724b0 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b75ffb700 1 -- 172.21.15.154:0/3499930979 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8b8001b9e0 con 0x7f8b840724b0 2024-01-16T17:10:32.858 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8410d3c0 con 0x7f8b840724b0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b75ffb700 1 -- 172.21.15.154:0/3499930979 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b8001a880 con 0x7f8b840724b0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.695+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8410d7d0 con 0x7f8b840724b0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.696+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b84045ad0 con 0x7f8b840724b0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.696+0000 7f8b75ffb700 1 -- 172.21.15.154:0/3499930979 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8b8000f070 con 0x7f8b840724b0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.697+0000 7f8b75ffb700 1 --2- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f8b6c02fd50 0x7f8b6c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.697+0000 7f8b75ffb700 1 -- 172.21.15.154:0/3499930979 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8b80041290 con 0x7f8b840724b0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.697+0000 7f8b88b2c700 1 --2- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f8b6c02fd50 0x7f8b6c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.697+0000 7f8b88b2c700 1 --2- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f8b6c02fd50 0x7f8b6c032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f8b780096a0 tx=0x7f8b78006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.701+0000 7f8b75ffb700 1 -- 172.21.15.154:0/3499930979 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b80019cd0 con 0x7f8b840724b0 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.853+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b8410e030 con 0x7f8b6c02fd50 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.854+0000 7f8b75ffb700 1 -- 172.21.15.154:0/3499930979 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f8b8410e030 con 0x7f8b6c02fd50 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f8b6c02fd50 msgr2=0x7f8b6c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 --2- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f8b6c02fd50 0x7f8b6c032210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f8b780096a0 tx=0x7f8b78006b40).stop 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 msgr2=0x7f8b8410ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 --2- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b8410ccd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f8b8000b190 tx=0x7f8b80006ee0).stop 2024-01-16T17:10:32.859 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 shutdown_connections 2024-01-16T17:10:32.860 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 --2- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f8b6c02fd50 0x7f8b6c032210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:32.860 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 --2- 172.21.15.154:0/3499930979 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b840724b0 0x7f8b8410ccd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:32.860 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 >> 172.21.15.154:0/3499930979 conn(0x7f8b840f74c0 msgr2=0x7f8b840f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:32.860 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 shutdown_connections 2024-01-16T17:10:32.860 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:32.855+0000 7f8b8b591700 1 -- 172.21.15.154:0/3499930979 wait complete. 2024-01-16T17:10:32.860 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVTdKBTk/l7yHsy1LAn3fkx1TBc5wMMM5me4zWd6oh8zvQaxKlzz+to1ciyb+hM5A52Tw7Ip0fkJRhhUGnuFHd90HtajdKJY0bz+W5+y1+7J4zPXnoR7p8hB77g0pPIfpojnrBl8c6ylc0Enx3Lu+m+pGB8U5KuFhRttwRttGm9OKK97TcW1F7CduC3UtoY1lGOoRimwVcx3GgJEeXcbeqinOEfxuazWeB1sDEDJqLIcymPyGsv3mAN3ulMxfRrXvO+h1pV/UPGiMneZmZSiiQ7WgTAjKYAzVbW1vcSxfAVZVSsPKGQYTobht5c/Fh5ijZAshwA6s+Zcx2n/UFgyPD3DeFYi+ahhApY/RS1GSKciLYTLn6+etkmS676cwlHPe1EsflDX1mu977o9uo9lkrr8ufSHQpSjtgr4sdARuuiPtc8+PvzkDLIwBqjwBCb0z+v7q9VYrAVuxJoieTCH8MPK0Yt1zOdHmNkt/sowTfM4XkLXahySuCYFgaloqe7uk= ceph-059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:10:34.113 INFO:teuthology.orchestra.run.smithi154.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-16T17:10:34.113 INFO:teuthology.orchestra.run.smithi154.stderr:Adding key to root@localhost authorized_keys... 2024-01-16T17:10:34.114 INFO:teuthology.orchestra.run.smithi154.stderr:Adding host smithi154... 2024-01-16T17:10:34.114 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi154 172.21.15.154 2024-01-16T17:10:34.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:34 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:32.855077+0000 mgr.smithi154.sllivz (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:34.502 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.519+0000 7fb49d209700 1 Processor -- start 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.519+0000 7fb49d209700 1 -- start start 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.519+0000 7fb49d209700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb4980fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.519+0000 7fb49d209700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980fee10 con 0x7fb4980703f0 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.519+0000 7fb496d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb4980fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.519+0000 7fb496d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb4980fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43400/0 (socket says 172.21.15.154:43400) 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.519+0000 7fb496d9d700 1 -- 172.21.15.154:0/2764254539 learned_addr learned my addr 172.21.15.154:0/2764254539 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb496d9d700 1 -- 172.21.15.154:0/2764254539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980fef50 con 0x7fb4980703f0 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb496d9d700 1 --2- 172.21.15.154:0/2764254539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb4980fe920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb48c00b5f0 tx=0x7fb48c006ee0).ready entity=mon.0 client_cookie=8a7e5e5b32d230e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:34.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb495d9b700 1 -- 172.21.15.154:0/2764254539 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb48c00d600 con 0x7fb4980703f0 2024-01-16T17:10:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb495d9b700 1 -- 172.21.15.154:0/2764254539 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb48c00dbd0 con 0x7fb4980703f0 2024-01-16T17:10:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb495d9b700 1 -- 172.21.15.154:0/2764254539 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb48c011c00 con 0x7fb4980703f0 2024-01-16T17:10:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb49d209700 1 -- 172.21.15.154:0/2764254539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 msgr2=0x7fb4980fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb49d209700 1 --2- 172.21.15.154:0/2764254539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb4980fe920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb48c00b5f0 tx=0x7fb48c006ee0).stop 2024-01-16T17:10:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb49d209700 1 -- 172.21.15.154:0/2764254539 shutdown_connections 2024-01-16T17:10:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.520+0000 7fb49d209700 1 --2- 172.21.15.154:0/2764254539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb4980fe920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb49d209700 1 -- 172.21.15.154:0/2764254539 >> 172.21.15.154:0/2764254539 conn(0x7fb4980f63c0 msgr2=0x7fb4980f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb49d209700 1 -- 172.21.15.154:0/2764254539 shutdown_connections 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb49d209700 1 -- 172.21.15.154:0/2764254539 wait complete. 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb49d209700 1 Processor -- start 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb49d209700 1 -- start start 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb49d209700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb49810a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb49d209700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49810aeb0 con 0x7fb4980703f0 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb496d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb49810a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.521+0000 7fb496d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb49810a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43404/0 (socket says 172.21.15.154:43404) 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.522+0000 7fb496d9d700 1 -- 172.21.15.154:0/532107039 learned_addr learned my addr 172.21.15.154:0/532107039 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.522+0000 7fb496d9d700 1 -- 172.21.15.154:0/532107039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb48c00b200 con 0x7fb4980703f0 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.522+0000 7fb496d9d700 1 --2- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb49810a9c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb48c00b190 tx=0x7fb48c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.522+0000 7fb47f7fe700 1 -- 172.21.15.154:0/532107039 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb48c01b410 con 0x7fb4980703f0 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.522+0000 7fb47f7fe700 1 -- 172.21.15.154:0/532107039 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fb48c01b9e0 con 0x7fb4980703f0 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.522+0000 7fb47f7fe700 1 -- 172.21.15.154:0/532107039 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb48c01a880 con 0x7fb4980703f0 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.522+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49810b0b0 con 0x7fb4980703f0 2024-01-16T17:10:34.506 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.522+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49810b4c0 con 0x7fb4980703f0 2024-01-16T17:10:34.507 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.523+0000 7fb47f7fe700 1 -- 172.21.15.154:0/532107039 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb48c00f070 con 0x7fb4980703f0 2024-01-16T17:10:34.507 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.523+0000 7fb47f7fe700 1 --2- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fb48402fdb0 0x7fb484032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:34.507 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.523+0000 7fb47f7fe700 1 -- 172.21.15.154:0/532107039 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb48c0411f0 con 0x7fb4980703f0 2024-01-16T17:10:34.507 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.523+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb498045ad0 con 0x7fb4980703f0 2024-01-16T17:10:34.507 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.523+0000 7fb49659c700 1 --2- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fb48402fdb0 0x7fb484032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:34.507 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.523+0000 7fb49659c700 1 --2- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fb48402fdb0 0x7fb484032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb4880096a0 tx=0x7fb488006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:34.509 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.527+0000 7fb47f7fe700 1 -- 172.21.15.154:0/532107039 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb48c016070 con 0x7fb4980703f0 2024-01-16T17:10:34.684 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:34.701+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi154", "addr": "172.21.15.154", "target": ["mon-mgr", ""]}) v1 -- 0x7fb498104900 con 0x7fb48402fdb0 2024-01-16T17:10:35.268 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Added host 'smithi154' 2024-01-16T17:10:35.268 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.285+0000 7fb47f7fe700 1 -- 172.21.15.154:0/532107039 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fb498104900 con 0x7fb48402fdb0 2024-01-16T17:10:35.269 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.287+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fb48402fdb0 msgr2=0x7fb484032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:35.269 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.287+0000 7fb49d209700 1 --2- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fb48402fdb0 0x7fb484032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb4880096a0 tx=0x7fb488006b40).stop 2024-01-16T17:10:35.269 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.288+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 msgr2=0x7fb49810a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:35.270 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.288+0000 7fb49d209700 1 --2- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb49810a9c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fb48c00b190 tx=0x7fb48c006ee0).stop 2024-01-16T17:10:35.270 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.288+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 shutdown_connections 2024-01-16T17:10:35.270 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.288+0000 7fb49d209700 1 --2- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7fb48402fdb0 0x7fb484032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:35.270 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.288+0000 7fb49d209700 1 --2- 172.21.15.154:0/532107039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4980703f0 0x7fb49810a9c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:35.270 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.288+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 >> 172.21.15.154:0/532107039 conn(0x7fb4980f63c0 msgr2=0x7fb4980f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:35.270 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.288+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 shutdown_connections 2024-01-16T17:10:35.270 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.288+0000 7fb49d209700 1 -- 172.21.15.154:0/532107039 wait complete. 2024-01-16T17:10:35.580 INFO:teuthology.orchestra.run.smithi154.stderr:Deploying mon service with default placement... 2024-01-16T17:10:35.580 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-16T17:10:35.899 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.917+0000 7efdba00c700 1 Processor -- start 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.917+0000 7efdba00c700 1 -- start start 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.917+0000 7efdba00c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.917+0000 7efdba00c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb40fb490 con 0x7efdb40fab80 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.918+0000 7efdb37fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.918+0000 7efdb37fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43410/0 (socket says 172.21.15.154:43410) 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.918+0000 7efdb37fe700 1 -- 172.21.15.154:0/2263168475 learned_addr learned my addr 172.21.15.154:0/2263168475 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.918+0000 7efdb37fe700 1 -- 172.21.15.154:0/2263168475 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb40fb5d0 con 0x7efdb40fab80 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.918+0000 7efdb37fe700 1 --2- 172.21.15.154:0/2263168475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb40fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7efda800b5f0 tx=0x7efda8006ee0).ready entity=mon.0 client_cookie=d13e5affe49ab51f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.918+0000 7efdb27fc700 1 -- 172.21.15.154:0/2263168475 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda800d600 con 0x7efdb40fab80 2024-01-16T17:10:35.902 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.918+0000 7efdb27fc700 1 -- 172.21.15.154:0/2263168475 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efda800dbd0 con 0x7efdb40fab80 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.918+0000 7efdb27fc700 1 -- 172.21.15.154:0/2263168475 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda8011c00 con 0x7efdb40fab80 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.919+0000 7efdba00c700 1 -- 172.21.15.154:0/2263168475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 msgr2=0x7efdb40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.919+0000 7efdba00c700 1 --2- 172.21.15.154:0/2263168475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb40fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7efda800b5f0 tx=0x7efda8006ee0).stop 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.919+0000 7efdba00c700 1 -- 172.21.15.154:0/2263168475 shutdown_connections 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.919+0000 7efdba00c700 1 --2- 172.21.15.154:0/2263168475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb40fafa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.919+0000 7efdba00c700 1 -- 172.21.15.154:0/2263168475 >> 172.21.15.154:0/2263168475 conn(0x7efdb40f63c0 msgr2=0x7efdb40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.919+0000 7efdba00c700 1 -- 172.21.15.154:0/2263168475 shutdown_connections 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.919+0000 7efdba00c700 1 -- 172.21.15.154:0/2263168475 wait complete. 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdba00c700 1 Processor -- start 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdba00c700 1 -- start start 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdba00c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb410aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdba00c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb410af10 con 0x7efdb40fab80 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdb37fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb410aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdb37fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb410aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43422/0 (socket says 172.21.15.154:43422) 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdb37fe700 1 -- 172.21.15.154:0/955157246 learned_addr learned my addr 172.21.15.154:0/955157246 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdb37fe700 1 -- 172.21.15.154:0/955157246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda800b200 con 0x7efdb40fab80 2024-01-16T17:10:35.903 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.920+0000 7efdb37fe700 1 --2- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb410aa20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7efda800e040 tx=0x7efda8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:35.904 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.921+0000 7efd9bfff700 1 -- 172.21.15.154:0/955157246 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda801b410 con 0x7efdb40fab80 2024-01-16T17:10:35.904 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.921+0000 7efd9bfff700 1 -- 172.21.15.154:0/955157246 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efda801b9e0 con 0x7efdb40fab80 2024-01-16T17:10:35.904 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.921+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb410b110 con 0x7efdb40fab80 2024-01-16T17:10:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.921+0000 7efd9bfff700 1 -- 172.21.15.154:0/955157246 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efda801a820 con 0x7efdb40fab80 2024-01-16T17:10:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.921+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb410b520 con 0x7efdb40fab80 2024-01-16T17:10:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.921+0000 7efd9bfff700 1 -- 172.21.15.154:0/955157246 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7efda800f070 con 0x7efdb40fab80 2024-01-16T17:10:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.921+0000 7efd9bfff700 1 --2- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efda002fd50 0x7efda0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.922+0000 7efd9bfff700 1 -- 172.21.15.154:0/955157246 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7efda8041450 con 0x7efdb40fab80 2024-01-16T17:10:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.922+0000 7efdb2ffd700 1 --2- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efda002fd50 0x7efda0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.922+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb4059100 con 0x7efdb40fab80 2024-01-16T17:10:35.905 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.922+0000 7efdb2ffd700 1 --2- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efda002fd50 0x7efda0032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7efda40096a0 tx=0x7efda4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:35.907 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:35.926+0000 7efd9bfff700 1 -- 172.21.15.154:0/955157246 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efda8016070 con 0x7efdb40fab80 2024-01-16T17:10:36.083 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.102+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7efdb4104a30 con 0x7efda002fd50 2024-01-16T17:10:36.087 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.106+0000 7efd9bfff700 1 -- 172.21.15.154:0/955157246 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7efdb4104a30 con 0x7efda002fd50 2024-01-16T17:10:36.087 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.108+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efda002fd50 msgr2=0x7efda0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.108+0000 7efdba00c700 1 --2- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efda002fd50 0x7efda0032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7efda40096a0 tx=0x7efda4006b40).stop 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.108+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 msgr2=0x7efdb410aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.108+0000 7efdba00c700 1 --2- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb410aa20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7efda800e040 tx=0x7efda8006ee0).stop 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.108+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 shutdown_connections 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.108+0000 7efdba00c700 1 --2- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efda002fd50 0x7efda0032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.108+0000 7efdba00c700 1 --2- 172.21.15.154:0/955157246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40fab80 0x7efdb410aa20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.108+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 >> 172.21.15.154:0/955157246 conn(0x7efdb40f63c0 msgr2=0x7efdb4070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.109+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 shutdown_connections 2024-01-16T17:10:36.090 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.109+0000 7efdba00c700 1 -- 172.21.15.154:0/955157246 wait complete. 2024-01-16T17:10:36.380 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:34.703249+0000 mgr.smithi154.sllivz (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "addr": "172.21.15.154", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:36.380 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:35.285083+0000 mon.smithi154 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:36.380 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:35.285387+0000 mgr.smithi154.sllivz (mgr.14114) 9 : cephadm [INF] Added host smithi154 2024-01-16T17:10:36.381 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:35.285677+0000 mon.smithi154 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:36.381 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:36.106481+0000 mon.smithi154 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:36.461 INFO:teuthology.orchestra.run.smithi154.stderr:Deploying mgr service with default placement... 2024-01-16T17:10:36.461 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-16T17:10:36.798 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.815+0000 7efcf2cfc700 1 Processor -- start 2024-01-16T17:10:36.798 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.815+0000 7efcf2cfc700 1 -- start start 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.815+0000 7efcf2cfc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.815+0000 7efcf2cfc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0fedd0 con 0x7efcec070430 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.815+0000 7efcf0a98700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.815+0000 7efcf0a98700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43426/0 (socket says 172.21.15.154:43426) 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.815+0000 7efcf0a98700 1 -- 172.21.15.154:0/2549587379 learned_addr learned my addr 172.21.15.154:0/2549587379 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.816+0000 7efcf0a98700 1 -- 172.21.15.154:0/2549587379 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0fef10 con 0x7efcec070430 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.816+0000 7efcf0a98700 1 --2- 172.21.15.154:0/2549587379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec0fe8e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efce000b5f0 tx=0x7efce0006ee0).ready entity=mon.0 client_cookie=c83640d822e0a44b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.816+0000 7efceb7fe700 1 -- 172.21.15.154:0/2549587379 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efce000d600 con 0x7efcec070430 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.816+0000 7efceb7fe700 1 -- 172.21.15.154:0/2549587379 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efce000dbd0 con 0x7efcec070430 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.816+0000 7efceb7fe700 1 -- 172.21.15.154:0/2549587379 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efce0011c00 con 0x7efcec070430 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.816+0000 7efcf2cfc700 1 -- 172.21.15.154:0/2549587379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 msgr2=0x7efcec0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.816+0000 7efcf2cfc700 1 --2- 172.21.15.154:0/2549587379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec0fe8e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efce000b5f0 tx=0x7efce0006ee0).stop 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.817+0000 7efcf2cfc700 1 -- 172.21.15.154:0/2549587379 shutdown_connections 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.817+0000 7efcf2cfc700 1 --2- 172.21.15.154:0/2549587379 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec0fe8e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:36.799 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.817+0000 7efcf2cfc700 1 -- 172.21.15.154:0/2549587379 >> 172.21.15.154:0/2549587379 conn(0x7efcec0f63c0 msgr2=0x7efcec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:36.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.817+0000 7efcf2cfc700 1 -- 172.21.15.154:0/2549587379 shutdown_connections 2024-01-16T17:10:36.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.817+0000 7efcf2cfc700 1 -- 172.21.15.154:0/2549587379 wait complete. 2024-01-16T17:10:36.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.817+0000 7efcf2cfc700 1 Processor -- start 2024-01-16T17:10:36.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.817+0000 7efcf2cfc700 1 -- start start 2024-01-16T17:10:36.800 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.818+0000 7efcf2cfc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.818+0000 7efcf2cfc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec10af30 con 0x7efcec070430 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.818+0000 7efcf0a98700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.818+0000 7efcf0a98700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43442/0 (socket says 172.21.15.154:43442) 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.818+0000 7efcf0a98700 1 -- 172.21.15.154:0/3665473977 learned_addr learned my addr 172.21.15.154:0/3665473977 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.818+0000 7efcf0a98700 1 -- 172.21.15.154:0/3665473977 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce000b200 con 0x7efcec070430 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.818+0000 7efcf0a98700 1 --2- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec10aa40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efce000e040 tx=0x7efce0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.818+0000 7efce97fa700 1 -- 172.21.15.154:0/3665473977 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efce000d9f0 con 0x7efcec070430 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.819+0000 7efce97fa700 1 -- 172.21.15.154:0/3665473977 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efce001a9f0 con 0x7efcec070430 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.819+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec10b130 con 0x7efcec070430 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.819+0000 7efce97fa700 1 -- 172.21.15.154:0/3665473977 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efce001b8c0 con 0x7efcec070430 2024-01-16T17:10:36.801 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.819+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec10b540 con 0x7efcec070430 2024-01-16T17:10:36.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.819+0000 7efce97fa700 1 -- 172.21.15.154:0/3665473977 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7efce000f070 con 0x7efcec070430 2024-01-16T17:10:36.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.819+0000 7efce97fa700 1 --2- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efcd002fdb0 0x7efcd0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:36.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.819+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcec045ad0 con 0x7efcec070430 2024-01-16T17:10:36.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.820+0000 7efce97fa700 1 -- 172.21.15.154:0/3665473977 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7efce00420c0 con 0x7efcec070430 2024-01-16T17:10:36.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.820+0000 7efcebfff700 1 --2- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efcd002fdb0 0x7efcd0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:36.802 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.820+0000 7efcebfff700 1 --2- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efcd002fdb0 0x7efcd0032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efcd80096a0 tx=0x7efcd8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:36.804 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.824+0000 7efce97fa700 1 -- 172.21.15.154:0/3665473977 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efce0016020 con 0x7efcec070430 2024-01-16T17:10:36.980 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:36.998+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7efcec10be20 con 0x7efcd002fdb0 2024-01-16T17:10:36.983 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.002+0000 7efce97fa700 1 -- 172.21.15.154:0/3665473977 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7efcec10be20 con 0x7efcd002fdb0 2024-01-16T17:10:36.983 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-16T17:10:36.986 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efcd002fdb0 msgr2=0x7efcd0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:36.986 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 --2- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efcd002fdb0 0x7efcd0032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7efcd80096a0 tx=0x7efcd8006b40).stop 2024-01-16T17:10:36.986 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 msgr2=0x7efcec10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:36.986 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 --2- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec10aa40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7efce000e040 tx=0x7efce0006ee0).stop 2024-01-16T17:10:36.986 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 shutdown_connections 2024-01-16T17:10:36.986 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 --2- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7efcd002fdb0 0x7efcd0032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:36.987 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 --2- 172.21.15.154:0/3665473977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efcec070430 0x7efcec10aa40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:36.987 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 >> 172.21.15.154:0/3665473977 conn(0x7efcec0f63c0 msgr2=0x7efcec0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:36.987 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 shutdown_connections 2024-01-16T17:10:36.987 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.004+0000 7efcf2cfc700 1 -- 172.21.15.154:0/3665473977 wait complete. 2024-01-16T17:10:37.361 INFO:teuthology.orchestra.run.smithi154.stderr:Deploying crash service with default placement... 2024-01-16T17:10:37.362 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-16T17:10:37.695 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.711+0000 7f667b59e700 1 Processor -- start 2024-01-16T17:10:37.695 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.711+0000 7f667b59e700 1 -- start start 2024-01-16T17:10:37.695 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.711+0000 7f667b59e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c100b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:37.695 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.711+0000 7f667b59e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0fc580 con 0x7f667c0703d0 2024-01-16T17:10:37.695 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.711+0000 7f667a59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:37.695 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.711+0000 7f667a59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c100b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43450/0 (socket says 172.21.15.154:43450) 2024-01-16T17:10:37.695 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.711+0000 7f667a59c700 1 -- 172.21.15.154:0/2695171942 learned_addr learned my addr 172.21.15.154:0/2695171942 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667a59c700 1 -- 172.21.15.154:0/2695171942 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c101080 con 0x7f667c0703d0 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667a59c700 1 --2- 172.21.15.154:0/2695171942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c100b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f666c00b5f0 tx=0x7f666c006ee0).ready entity=mon.0 client_cookie=15f69e3e8c4411bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667959a700 1 -- 172.21.15.154:0/2695171942 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f666c00d600 con 0x7f667c0703d0 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667959a700 1 -- 172.21.15.154:0/2695171942 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f666c00dbd0 con 0x7f667c0703d0 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667959a700 1 -- 172.21.15.154:0/2695171942 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f666c011c00 con 0x7f667c0703d0 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667b59e700 1 -- 172.21.15.154:0/2695171942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 msgr2=0x7f667c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667b59e700 1 --2- 172.21.15.154:0/2695171942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c100b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f666c00b5f0 tx=0x7f666c006ee0).stop 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667b59e700 1 -- 172.21.15.154:0/2695171942 shutdown_connections 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667b59e700 1 --2- 172.21.15.154:0/2695171942 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c100b90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667b59e700 1 -- 172.21.15.154:0/2695171942 >> 172.21.15.154:0/2695171942 conn(0x7f667c0f6210 msgr2=0x7f667c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.712+0000 7f667b59e700 1 -- 172.21.15.154:0/2695171942 shutdown_connections 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.713+0000 7f667b59e700 1 -- 172.21.15.154:0/2695171942 wait complete. 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.713+0000 7f667b59e700 1 Processor -- start 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.713+0000 7f667b59e700 1 -- start start 2024-01-16T17:10:37.696 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.713+0000 7f667b59e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c10ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.713+0000 7f667b59e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f667c10cf70 con 0x7f667c0703d0 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.713+0000 7f667a59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c10ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.713+0000 7f667a59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c10ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43460/0 (socket says 172.21.15.154:43460) 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.713+0000 7f667a59c700 1 -- 172.21.15.154:0/2206429715 learned_addr learned my addr 172.21.15.154:0/2206429715 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.714+0000 7f667a59c700 1 -- 172.21.15.154:0/2206429715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666c00b200 con 0x7f667c0703d0 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.714+0000 7f667a59c700 1 --2- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c10ca80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f666c00e040 tx=0x7f666c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.714+0000 7f666affd700 1 -- 172.21.15.154:0/2206429715 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f666c00d9f0 con 0x7f667c0703d0 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.714+0000 7f666affd700 1 -- 172.21.15.154:0/2206429715 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f666c01a9f0 con 0x7f667c0703d0 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.714+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c10d170 con 0x7f667c0703d0 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.714+0000 7f666affd700 1 -- 172.21.15.154:0/2206429715 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f666c01b8c0 con 0x7f667c0703d0 2024-01-16T17:10:37.697 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.714+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c10d580 con 0x7f667c0703d0 2024-01-16T17:10:37.698 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.715+0000 7f666affd700 1 -- 172.21.15.154:0/2206429715 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f666c00f070 con 0x7f667c0703d0 2024-01-16T17:10:37.698 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.715+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f667c045ad0 con 0x7f667c0703d0 2024-01-16T17:10:37.698 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.715+0000 7f666affd700 1 --2- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f666002fd50 0x7f6660032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:37.698 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.715+0000 7f666affd700 1 -- 172.21.15.154:0/2206429715 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f666c0413e0 con 0x7f667c0703d0 2024-01-16T17:10:37.698 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.715+0000 7f6679d9b700 1 --2- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f666002fd50 0x7f6660032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:37.698 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.715+0000 7f6679d9b700 1 --2- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f666002fd50 0x7f6660032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f66700096a0 tx=0x7f6670006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:37.700 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.719+0000 7f666affd700 1 -- 172.21.15.154:0/2206429715 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f666c0224f0 con 0x7f667c0703d0 2024-01-16T17:10:37.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:36.103502+0000 mgr.smithi154.sllivz (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:37.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:36.104187+0000 mgr.smithi154.sllivz (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-16T17:10:37.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:37.002695+0000 mon.smithi154 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:37.875 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.894+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f667c10dec0 con 0x7f666002fd50 2024-01-16T17:10:37.879 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.898+0000 7f666affd700 1 -- 172.21.15.154:0/2206429715 <== mgr.14114 v2:172.21.15.154:6800/3611230081 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f667c10dec0 con 0x7f666002fd50 2024-01-16T17:10:37.879 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-16T17:10:37.881 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.899+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f666002fd50 msgr2=0x7f6660032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:37.881 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.899+0000 7f667b59e700 1 --2- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f666002fd50 0x7f6660032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f66700096a0 tx=0x7f6670006b40).stop 2024-01-16T17:10:37.881 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.899+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 msgr2=0x7f667c10ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:37.881 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.899+0000 7f667b59e700 1 --2- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c10ca80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f666c00e040 tx=0x7f666c006ee0).stop 2024-01-16T17:10:37.881 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.899+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 shutdown_connections 2024-01-16T17:10:37.882 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.899+0000 7f667b59e700 1 --2- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f666002fd50 0x7f6660032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:37.882 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.899+0000 7f667b59e700 1 --2- 172.21.15.154:0/2206429715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c0703d0 0x7f667c10ca80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:37.882 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.899+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 >> 172.21.15.154:0/2206429715 conn(0x7f667c0f6210 msgr2=0x7f667c0f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:37.882 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.900+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 shutdown_connections 2024-01-16T17:10:37.882 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:37.900+0000 7f667b59e700 1 -- 172.21.15.154:0/2206429715 wait complete. 2024-01-16T17:10:38.262 INFO:teuthology.orchestra.run.smithi154.stderr:Enabling mgr prometheus module... 2024-01-16T17:10:38.262 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-16T17:10:38.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:38 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:36.999787+0000 mgr.smithi154.sllivz (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:38.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:38 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:37.000315+0000 mgr.smithi154.sllivz (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-16T17:10:38.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:38 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:37.897946+0000 mon.smithi154 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.154:0/4220626083' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:38.581 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.600+0000 7f4b8e90a700 1 Processor -- start 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.600+0000 7f4b8e90a700 1 -- start start 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.600+0000 7f4b8e90a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b880fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.600+0000 7f4b8e90a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b880fedd0 con 0x7f4b88070430 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.601+0000 7f4b87fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b880fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.601+0000 7f4b87fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b880fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43464/0 (socket says 172.21.15.154:43464) 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.601+0000 7f4b87fff700 1 -- 172.21.15.154:0/3851024058 learned_addr learned my addr 172.21.15.154:0/3851024058 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.601+0000 7f4b87fff700 1 -- 172.21.15.154:0/3851024058 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b880fef10 con 0x7f4b88070430 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.601+0000 7f4b87fff700 1 --2- 172.21.15.154:0/3851024058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b880fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f4b7c00b5f0 tx=0x7f4b7c006ee0).ready entity=mon.0 client_cookie=539f0a750e544131 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.601+0000 7f4b86ffd700 1 -- 172.21.15.154:0/3851024058 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b7c00d600 con 0x7f4b88070430 2024-01-16T17:10:38.583 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.601+0000 7f4b86ffd700 1 -- 172.21.15.154:0/3851024058 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4b7c00dbd0 con 0x7f4b88070430 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.601+0000 7f4b86ffd700 1 -- 172.21.15.154:0/3851024058 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b7c011c00 con 0x7f4b88070430 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.602+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3851024058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 msgr2=0x7f4b880fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.602+0000 7f4b8e90a700 1 --2- 172.21.15.154:0/3851024058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b880fe8e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f4b7c00b5f0 tx=0x7f4b7c006ee0).stop 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.602+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3851024058 shutdown_connections 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.602+0000 7f4b8e90a700 1 --2- 172.21.15.154:0/3851024058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b880fe8e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.602+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3851024058 >> 172.21.15.154:0/3851024058 conn(0x7f4b880f63c0 msgr2=0x7f4b880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.602+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3851024058 shutdown_connections 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.602+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3851024058 wait complete. 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.603+0000 7f4b8e90a700 1 Processor -- start 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.603+0000 7f4b8e90a700 1 -- start start 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.603+0000 7f4b8e90a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b8810aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:38.585 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.603+0000 7f4b8e90a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8810aef0 con 0x7f4b88070430 2024-01-16T17:10:38.586 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.603+0000 7f4b87fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b8810aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:38.586 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.603+0000 7f4b87fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b8810aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43470/0 (socket says 172.21.15.154:43470) 2024-01-16T17:10:38.586 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.603+0000 7f4b87fff700 1 -- 172.21.15.154:0/3992383341 learned_addr learned my addr 172.21.15.154:0/3992383341 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.604+0000 7f4b87fff700 1 -- 172.21.15.154:0/3992383341 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c00b200 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.604+0000 7f4b87fff700 1 --2- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b8810aa00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f4b7c00b190 tx=0x7f4b7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.604+0000 7f4b84ff9700 1 -- 172.21.15.154:0/3992383341 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b7c01b410 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.604+0000 7f4b84ff9700 1 -- 172.21.15.154:0/3992383341 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4b7c01b9e0 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.604+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8810b0f0 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.604+0000 7f4b84ff9700 1 -- 172.21.15.154:0/3992383341 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b7c01a880 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.604+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8810b500 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.605+0000 7f4b84ff9700 1 -- 172.21.15.154:0/3992383341 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4b7c00f070 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.605+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b88045ad0 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.605+0000 7f4b84ff9700 1 --2- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f4b7402fd50 0x7f4b74032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.605+0000 7f4b84ff9700 1 -- 172.21.15.154:0/3992383341 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4b7c041280 con 0x7f4b88070430 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.605+0000 7f4b877fe700 1 --2- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f4b7402fd50 0x7f4b74032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:38.587 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.605+0000 7f4b877fe700 1 --2- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f4b7402fd50 0x7f4b74032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4b780096a0 tx=0x7f4b78006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:38.590 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.609+0000 7f4b84ff9700 1 -- 172.21.15.154:0/3992383341 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b7c010360 con 0x7f4b88070430 2024-01-16T17:10:38.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:38.796+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f4b880ffb90 con 0x7f4b88070430 2024-01-16T17:10:39.361 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.380+0000 7f4b84ff9700 1 -- 172.21.15.154:0/3992383341 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f4b7c025980 con 0x7f4b88070430 2024-01-16T17:10:39.362 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.381+0000 7f4b84ff9700 1 -- 172.21.15.154:0/3992383341 <== mon.0 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f4b7c005700 con 0x7f4b88070430 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f4b7402fd50 msgr2=0x7f4b74032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 --2- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f4b7402fd50 0x7f4b74032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4b780096a0 tx=0x7f4b78006b40).stop 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 msgr2=0x7f4b8810aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 --2- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b8810aa00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f4b7c00b190 tx=0x7f4b7c006ee0).stop 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 shutdown_connections 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 --2- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f4b7402fd50 0x7f4b74032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 --2- 172.21.15.154:0/3992383341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b88070430 0x7f4b8810aa00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 >> 172.21.15.154:0/3992383341 conn(0x7f4b880f63c0 msgr2=0x7f4b880f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 shutdown_connections 2024-01-16T17:10:39.365 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:39.383+0000 7f4b8e90a700 1 -- 172.21.15.154:0/3992383341 wait complete. 2024-01-16T17:10:39.647 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:37.895139+0000 mgr.smithi154.sllivz (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:39.647 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:37.895743+0000 mgr.smithi154.sllivz (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-16T17:10:39.647 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:38.797228+0000 mon.smithi154 (mon.0) 58 : audit [INF] from='client.? 172.21.15.154:0/3992383341' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-16T17:10:39.695 INFO:teuthology.orchestra.run.smithi154.stderr:Deploying prometheus service with default placement... 2024-01-16T17:10:39.695 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.091+0000 7f33b9279700 1 Processor -- start 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.091+0000 7f33b9279700 1 -- start start 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.091+0000 7f33b9279700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b40fae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.091+0000 7f33b9279700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b40fb310 con 0x7f33b40faa00 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.091+0000 7f33b2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b40fae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.091+0000 7f33b2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b40fae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43492/0 (socket says 172.21.15.154:43492) 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.091+0000 7f33b2ffd700 1 -- 172.21.15.154:0/3177951361 learned_addr learned my addr 172.21.15.154:0/3177951361 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.091+0000 7f33b2ffd700 1 -- 172.21.15.154:0/3177951361 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b40fb450 con 0x7f33b40faa00 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b2ffd700 1 --2- 172.21.15.154:0/3177951361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b40fae20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f33a400b5f0 tx=0x7f33a4006ee0).ready entity=mon.0 client_cookie=b66f24a99be61dd1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b1ffb700 1 -- 172.21.15.154:0/3177951361 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33a400d600 con 0x7f33b40faa00 2024-01-16T17:10:40.074 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b1ffb700 1 -- 172.21.15.154:0/3177951361 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f33a400dbd0 con 0x7f33b40faa00 2024-01-16T17:10:40.075 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b1ffb700 1 -- 172.21.15.154:0/3177951361 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33a4011c00 con 0x7f33b40faa00 2024-01-16T17:10:40.075 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b9279700 1 -- 172.21.15.154:0/3177951361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 msgr2=0x7f33b40fae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:40.075 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b9279700 1 --2- 172.21.15.154:0/3177951361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b40fae20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f33a400b5f0 tx=0x7f33a4006ee0).stop 2024-01-16T17:10:40.075 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b9279700 1 -- 172.21.15.154:0/3177951361 shutdown_connections 2024-01-16T17:10:40.075 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b9279700 1 --2- 172.21.15.154:0/3177951361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b40fae20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:40.075 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.092+0000 7f33b9279700 1 -- 172.21.15.154:0/3177951361 >> 172.21.15.154:0/3177951361 conn(0x7f33b40f6240 msgr2=0x7f33b40f8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:40.075 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.093+0000 7f33b9279700 1 -- 172.21.15.154:0/3177951361 shutdown_connections 2024-01-16T17:10:40.075 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.093+0000 7f33b9279700 1 -- 172.21.15.154:0/3177951361 wait complete. 2024-01-16T17:10:40.077 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.093+0000 7f33b9279700 1 Processor -- start 2024-01-16T17:10:40.077 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.093+0000 7f33b9279700 1 -- start start 2024-01-16T17:10:40.077 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.093+0000 7f33b9279700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b4108670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:40.077 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.093+0000 7f33b9279700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33b4108b60 con 0x7f33b40faa00 2024-01-16T17:10:40.077 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f33b2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b4108670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:40.077 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f33b2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b4108670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:43504/0 (socket says 172.21.15.154:43504) 2024-01-16T17:10:40.077 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f33b2ffd700 1 -- 172.21.15.154:0/3729944609 learned_addr learned my addr 172.21.15.154:0/3729944609 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f33b2ffd700 1 -- 172.21.15.154:0/3729944609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a400b200 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f33b2ffd700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b4108670 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f33a400e040 tx=0x7f33a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33a401b410 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33b4108d60 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f33a401b9e0 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33a401a820 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.094+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33b4109170 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.095+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33b4045ad0 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.095+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f33a400f070 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.095+0000 7f339b7fe700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 0x7f33a00344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.095+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f33a40413d0 con 0x7f33b40faa00 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.095+0000 7f33b27fc700 1 -- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 msgr2=0x7f33a00344b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/3611230081 2024-01-16T17:10:40.078 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.095+0000 7f33b27fc700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 0x7f33a00344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-16T17:10:40.080 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.099+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33a4022310 con 0x7f33b40faa00 2024-01-16T17:10:40.264 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.283+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 --> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f33b4102590 con 0x7f33a0031ff0 2024-01-16T17:10:40.277 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.296+0000 7f33b27fc700 1 -- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 msgr2=0x7f33a00344b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/3611230081 2024-01-16T17:10:40.277 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.296+0000 7f33b27fc700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 0x7f33a00344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-16T17:10:40.678 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.697+0000 7f33b27fc700 1 -- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 msgr2=0x7f33a00344b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/3611230081 2024-01-16T17:10:40.678 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:40.697+0000 7f33b27fc700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 0x7f33a00344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-16T17:10:40.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:40 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:39.381414+0000 mon.smithi154 (mon.0) 59 : audit [INF] from='client.? 172.21.15.154:0/3992383341' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-16T17:10:40.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:40 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:39.381465+0000 mon.smithi154 (mon.0) 60 : cluster [DBG] mgrmap e8: smithi154.sllivz(active, since 16s) 2024-01-16T17:10:41.479 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:41.498+0000 7f33b27fc700 1 -- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 msgr2=0x7f33a00344b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/3611230081 2024-01-16T17:10:41.479 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:41.498+0000 7f33b27fc700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 0x7f33a00344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-16T17:10:43.080 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:43.099+0000 7f33b27fc700 1 -- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 msgr2=0x7f33a00344b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/3611230081 2024-01-16T17:10:43.080 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:43.099+0000 7f33b27fc700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 0x7f33a00344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-16T17:10:44.359 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:44.376+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f33a401aa10 con 0x7f33b40faa00 2024-01-16T17:10:44.359 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:44.376+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 msgr2=0x7f33a00344b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T17:10:44.359 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:44.376+0000 7f339b7fe700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 0x7f33a00344b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:45.361 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.377+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mon.0 v2:172.21.15.154:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f33a403d3b0 con 0x7f33b40faa00 2024-01-16T17:10:45.361 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.377+0000 7f339b7fe700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f33a0035e60 0x7f33a0038250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:45.361 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.377+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 --> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f33b4102590 con 0x7f33a0035e60 2024-01-16T17:10:45.361 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.378+0000 7f33b27fc700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f33a0035e60 0x7f33a0038250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:45.361 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.378+0000 7f33b27fc700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f33a0035e60 0x7f33a0038250 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f33a80036d0 tx=0x7f33a8006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:45.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.386+0000 7f339b7fe700 1 -- 172.21.15.154:0/3729944609 <== mgr.14142 v2:172.21.15.154:6800/1485441924 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f33b4102590 con 0x7f33a0035e60 2024-01-16T17:10:45.368 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f33a0035e60 msgr2=0x7f33a0038250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f33a0035e60 0x7f33a0038250 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f33a80036d0 tx=0x7f33a8006cb0).stop 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 msgr2=0x7f33b4108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b4108670 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f33a400e040 tx=0x7f33a4006ee0).stop 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 shutdown_connections 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f33b40faa00 0x7f33b4108670 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f33a0035e60 0x7f33a0038250 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 --2- 172.21.15.154:0/3729944609 >> [v2:172.21.15.154:6800/3611230081,v1:172.21.15.154:6801/3611230081] conn(0x7f33a0031ff0 0x7f33a00344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 >> 172.21.15.154:0/3729944609 conn(0x7f33b40f6240 msgr2=0x7f33b40ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 shutdown_connections 2024-01-16T17:10:45.371 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:45.388+0000 7f33b9279700 1 -- 172.21.15.154:0/3729944609 wait complete. 2024-01-16T17:10:45.654 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:44.367090+0000 mon.smithi154 (mon.0) 61 : cluster [INF] Active manager daemon smithi154.sllivz restarted 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:44.367519+0000 mon.smithi154 (mon.0) 62 : cluster [INF] Activating manager daemon smithi154.sllivz 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:44.369134+0000 mon.smithi154 (mon.0) 63 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:44.377872+0000 mon.smithi154 (mon.0) 64 : cluster [DBG] mgrmap e9: smithi154.sllivz(active, starting, since 0.0104597s) 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.379004+0000 mon.smithi154 (mon.0) 65 : audit [DBG] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.379549+0000 mon.smithi154 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mgr metadata", "who": "smithi154.sllivz", "id": "smithi154.sllivz"}]: dispatch 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.380203+0000 mon.smithi154 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.380362+0000 mon.smithi154 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.380485+0000 mon.smithi154 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:44.394951+0000 mon.smithi154 (mon.0) 70 : cluster [INF] Manager daemon smithi154.sllivz is now available 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.492123+0000 mon.smithi154 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.493366+0000 mon.smithi154 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.507502+0000 mon.smithi154 (mon.0) 73 : audit [INF] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:10:45.655 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:44.510161+0000 mon.smithi154 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:10:45.710 INFO:teuthology.orchestra.run.smithi154.stderr:Deploying grafana service with default placement... 2024-01-16T17:10:45.710 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-16T17:10:46.040 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.057+0000 7f78a8573700 1 Processor -- start 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.057+0000 7f78a8573700 1 -- start start 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.057+0000 7f78a8573700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.057+0000 7f78a8573700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00fb490 con 0x7f78a00fab80 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.058+0000 7f78a630f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.058+0000 7f78a630f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52348/0 (socket says 172.21.15.154:52348) 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.058+0000 7f78a630f700 1 -- 172.21.15.154:0/1243058117 learned_addr learned my addr 172.21.15.154:0/1243058117 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.058+0000 7f78a630f700 1 -- 172.21.15.154:0/1243058117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00fb5d0 con 0x7f78a00fab80 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.058+0000 7f78a630f700 1 --2- 172.21.15.154:0/1243058117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a00fafa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f789000b5f0 tx=0x7f7890006ee0).ready entity=mon.0 client_cookie=1a39c41224088167 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.058+0000 7f78a530d700 1 -- 172.21.15.154:0/1243058117 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f789000d600 con 0x7f78a00fab80 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.058+0000 7f78a530d700 1 -- 172.21.15.154:0/1243058117 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f789000dbd0 con 0x7f78a00fab80 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.058+0000 7f78a530d700 1 -- 172.21.15.154:0/1243058117 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7890011c00 con 0x7f78a00fab80 2024-01-16T17:10:46.043 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 -- 172.21.15.154:0/1243058117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 msgr2=0x7f78a00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 --2- 172.21.15.154:0/1243058117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a00fafa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f789000b5f0 tx=0x7f7890006ee0).stop 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 -- 172.21.15.154:0/1243058117 shutdown_connections 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 --2- 172.21.15.154:0/1243058117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a00fafa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 -- 172.21.15.154:0/1243058117 >> 172.21.15.154:0/1243058117 conn(0x7f78a00f63c0 msgr2=0x7f78a00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 -- 172.21.15.154:0/1243058117 shutdown_connections 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 -- 172.21.15.154:0/1243058117 wait complete. 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 Processor -- start 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.059+0000 7f78a8573700 1 -- start start 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a8573700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a010cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a8573700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a010d0d0 con 0x7f78a00fab80 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a630f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a010cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a630f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a010cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52360/0 (socket says 172.21.15.154:52360) 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a630f700 1 -- 172.21.15.154:0/1484507704 learned_addr learned my addr 172.21.15.154:0/1484507704 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a630f700 1 -- 172.21.15.154:0/1484507704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789000b200 con 0x7f78a00fab80 2024-01-16T17:10:46.044 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a630f700 1 --2- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a010cbe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f789000b1d0 tx=0x7f7890006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:46.045 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f788effd700 1 -- 172.21.15.154:0/1484507704 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f789001b410 con 0x7f78a00fab80 2024-01-16T17:10:46.045 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f788effd700 1 -- 172.21.15.154:0/1484507704 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f789001b9e0 con 0x7f78a00fab80 2024-01-16T17:10:46.045 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a010d2d0 con 0x7f78a00fab80 2024-01-16T17:10:46.045 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f788effd700 1 -- 172.21.15.154:0/1484507704 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f789001a840 con 0x7f78a00fab80 2024-01-16T17:10:46.045 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.060+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a010d6e0 con 0x7f78a00fab80 2024-01-16T17:10:46.046 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.061+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a0045ad0 con 0x7f78a00fab80 2024-01-16T17:10:46.046 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.061+0000 7f788effd700 1 -- 172.21.15.154:0/1484507704 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f789000f070 con 0x7f78a00fab80 2024-01-16T17:10:46.046 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.062+0000 7f788effd700 1 --2- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f788802fd40 0x7f7888032200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:46.046 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.062+0000 7f788effd700 1 -- 172.21.15.154:0/1484507704 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f78900411b0 con 0x7f78a00fab80 2024-01-16T17:10:46.046 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.062+0000 7f78a5b0e700 1 --2- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f788802fd40 0x7f7888032200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:46.046 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.062+0000 7f78a5b0e700 1 --2- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f788802fd40 0x7f7888032200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f789c0096a0 tx=0x7f789c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:46.048 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.066+0000 7f788effd700 1 -- 172.21.15.154:0/1484507704 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7890019950 con 0x7f78a00fab80 2024-01-16T17:10:46.222 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.240+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 --> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f78a010dfc0 con 0x7f788802fd40 2024-01-16T17:10:46.226 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.244+0000 7f788effd700 1 -- 172.21.15.154:0/1484507704 <== mgr.14142 v2:172.21.15.154:6800/1485441924 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f78a010dfc0 con 0x7f788802fd40 2024-01-16T17:10:46.226 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-16T17:10:46.228 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.245+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f788802fd40 msgr2=0x7f7888032200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:46.228 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.245+0000 7f78a8573700 1 --2- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f788802fd40 0x7f7888032200 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f789c0096a0 tx=0x7f789c006b40).stop 2024-01-16T17:10:46.228 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.245+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 msgr2=0x7f78a010cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:46.228 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.245+0000 7f78a8573700 1 --2- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a010cbe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f789000b1d0 tx=0x7f7890006ee0).stop 2024-01-16T17:10:46.228 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.245+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 shutdown_connections 2024-01-16T17:10:46.229 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.245+0000 7f78a8573700 1 --2- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f788802fd40 0x7f7888032200 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:46.229 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.245+0000 7f78a8573700 1 --2- 172.21.15.154:0/1484507704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a00fab80 0x7f78a010cbe0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:46.229 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.245+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 >> 172.21.15.154:0/1484507704 conn(0x7f78a00f63c0 msgr2=0x7f78a00ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:46.229 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.246+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 shutdown_connections 2024-01-16T17:10:46.229 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.246+0000 7f78a8573700 1 -- 172.21.15.154:0/1484507704 wait complete. 2024-01-16T17:10:46.504 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:46 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:45.380864+0000 mon.smithi154 (mon.0) 75 : cluster [DBG] mgrmap e10: smithi154.sllivz(active, since 1.01345s) 2024-01-16T17:10:46.504 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:46 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:45.381966+0000 mgr.smithi154.sllivz (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-16T17:10:46.504 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:46 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:45.386808+0000 mon.smithi154 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:46.504 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:46 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:46.244976+0000 mon.smithi154 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:46.576 INFO:teuthology.orchestra.run.smithi154.stderr:Deploying node-exporter service with default placement... 2024-01-16T17:10:46.577 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-16T17:10:46.908 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe602011700 1 Processor -- start 2024-01-16T17:10:46.908 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe602011700 1 -- start start 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe602011700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe602011700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0fc730 con 0x7fe5fc0703d0 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe5fb7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe5fb7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52372/0 (socket says 172.21.15.154:52372) 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe5fb7fe700 1 -- 172.21.15.154:0/2414930149 learned_addr learned my addr 172.21.15.154:0/2414930149 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe5fb7fe700 1 -- 172.21.15.154:0/2414930149 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc101230 con 0x7fe5fc0703d0 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.925+0000 7fe5fb7fe700 1 --2- 172.21.15.154:0/2414930149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc100d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe5ec00b5f0 tx=0x7fe5ec006ee0).ready entity=mon.0 client_cookie=1113ad00813796d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe5fa7fc700 1 -- 172.21.15.154:0/2414930149 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5ec00d600 con 0x7fe5fc0703d0 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe5fa7fc700 1 -- 172.21.15.154:0/2414930149 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe5ec00dbd0 con 0x7fe5fc0703d0 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe5fa7fc700 1 -- 172.21.15.154:0/2414930149 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5ec011c00 con 0x7fe5fc0703d0 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe602011700 1 -- 172.21.15.154:0/2414930149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 msgr2=0x7fe5fc100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe602011700 1 --2- 172.21.15.154:0/2414930149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc100d40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe5ec00b5f0 tx=0x7fe5ec006ee0).stop 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe602011700 1 -- 172.21.15.154:0/2414930149 shutdown_connections 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe602011700 1 --2- 172.21.15.154:0/2414930149 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc100d40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe602011700 1 -- 172.21.15.154:0/2414930149 >> 172.21.15.154:0/2414930149 conn(0x7fe5fc0f63c0 msgr2=0x7fe5fc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:46.909 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe602011700 1 -- 172.21.15.154:0/2414930149 shutdown_connections 2024-01-16T17:10:46.910 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.926+0000 7fe602011700 1 -- 172.21.15.154:0/2414930149 wait complete. 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.927+0000 7fe602011700 1 Processor -- start 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.927+0000 7fe602011700 1 -- start start 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.927+0000 7fe602011700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.927+0000 7fe602011700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc10d100 con 0x7fe5fc0703d0 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.927+0000 7fe5fb7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.927+0000 7fe5fb7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52384/0 (socket says 172.21.15.154:52384) 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.927+0000 7fe5fb7fe700 1 -- 172.21.15.154:0/4210257047 learned_addr learned my addr 172.21.15.154:0/4210257047 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.927+0000 7fe5fb7fe700 1 -- 172.21.15.154:0/4210257047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec00b200 con 0x7fe5fc0703d0 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.928+0000 7fe5fb7fe700 1 --2- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc10cc10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe5ec00e040 tx=0x7fe5ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.928+0000 7fe5e3fff700 1 -- 172.21.15.154:0/4210257047 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5ec00d9f0 con 0x7fe5fc0703d0 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.928+0000 7fe5e3fff700 1 -- 172.21.15.154:0/4210257047 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe5ec01a9f0 con 0x7fe5fc0703d0 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.928+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc10d300 con 0x7fe5fc0703d0 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.928+0000 7fe5e3fff700 1 -- 172.21.15.154:0/4210257047 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5ec01b8c0 con 0x7fe5fc0703d0 2024-01-16T17:10:46.912 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.928+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc10d710 con 0x7fe5fc0703d0 2024-01-16T17:10:46.913 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.928+0000 7fe5e3fff700 1 -- 172.21.15.154:0/4210257047 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fe5ec00f070 con 0x7fe5fc0703d0 2024-01-16T17:10:46.913 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.929+0000 7fe5e3fff700 1 --2- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fe5e802fd70 0x7fe5e8032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:46.913 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.929+0000 7fe5e3fff700 1 -- 172.21.15.154:0/4210257047 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe5ec0412b0 con 0x7fe5fc0703d0 2024-01-16T17:10:46.913 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.929+0000 7fe5faffd700 1 --2- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fe5e802fd70 0x7fe5e8032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:46.913 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.929+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc059100 con 0x7fe5fc0703d0 2024-01-16T17:10:46.913 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.929+0000 7fe5faffd700 1 --2- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fe5e802fd70 0x7fe5e8032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe5f00096a0 tx=0x7fe5f0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:46.915 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:46.933+0000 7fe5e3fff700 1 -- 172.21.15.154:0/4210257047 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5ec016070 con 0x7fe5fc0703d0 2024-01-16T17:10:47.093 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.111+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 --> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5fc106a60 con 0x7fe5e802fd70 2024-01-16T17:10:47.097 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.115+0000 7fe5e3fff700 1 -- 172.21.15.154:0/4210257047 <== mgr.14142 v2:172.21.15.154:6800/1485441924 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe5fc106a60 con 0x7fe5e802fd70 2024-01-16T17:10:47.097 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-16T17:10:47.099 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fe5e802fd70 msgr2=0x7fe5e8032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:47.099 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 --2- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fe5e802fd70 0x7fe5e8032230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe5f00096a0 tx=0x7fe5f0006b40).stop 2024-01-16T17:10:47.099 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 msgr2=0x7fe5fc10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:47.099 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 --2- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc10cc10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe5ec00e040 tx=0x7fe5ec006ee0).stop 2024-01-16T17:10:47.100 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 shutdown_connections 2024-01-16T17:10:47.100 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 --2- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fe5e802fd70 0x7fe5e8032230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:47.100 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 --2- 172.21.15.154:0/4210257047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5fc0703d0 0x7fe5fc10cc10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:47.100 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 >> 172.21.15.154:0/4210257047 conn(0x7fe5fc0f63c0 msgr2=0x7fe5fc0f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:47.100 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 shutdown_connections 2024-01-16T17:10:47.100 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.117+0000 7fe602011700 1 -- 172.21.15.154:0/4210257047 wait complete. 2024-01-16T17:10:47.435 INFO:teuthology.orchestra.run.smithi154.stderr:Deploying alertmanager service with default placement... 2024-01-16T17:10:47.436 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-16T17:10:47.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:46.242257+0000 mgr.smithi154.sllivz (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:47.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:46.242737+0000 mgr.smithi154.sllivz (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-16T17:10:47.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:47.116167+0000 mon.smithi154 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:47.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:47.248883+0000 mon.smithi154 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi154.sllivz(active, since 2s) 2024-01-16T17:10:47.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.790+0000 7f2d4a7d0700 1 Processor -- start 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.790+0000 7f2d4a7d0700 1 -- start start 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.790+0000 7f2d4a7d0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d44100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.790+0000 7f2d4a7d0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d440fc710 con 0x7f2d440703f0 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.791+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d44100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.791+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d44100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52394/0 (socket says 172.21.15.154:52394) 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.791+0000 7f2d43fff700 1 -- 172.21.15.154:0/2991804108 learned_addr learned my addr 172.21.15.154:0/2991804108 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.791+0000 7f2d43fff700 1 -- 172.21.15.154:0/2991804108 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44101210 con 0x7f2d440703f0 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.791+0000 7f2d43fff700 1 --2- 172.21.15.154:0/2991804108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d44100d20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f2d3800b5f0 tx=0x7f2d38006ee0).ready entity=mon.0 client_cookie=d3ae554066080fc4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:47.774 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.791+0000 7f2d42ffd700 1 -- 172.21.15.154:0/2991804108 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d3800d600 con 0x7f2d440703f0 2024-01-16T17:10:47.776 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.791+0000 7f2d42ffd700 1 -- 172.21.15.154:0/2991804108 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2d3800dbd0 con 0x7f2d440703f0 2024-01-16T17:10:47.776 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.791+0000 7f2d42ffd700 1 -- 172.21.15.154:0/2991804108 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d38011c00 con 0x7f2d440703f0 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.792+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/2991804108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 msgr2=0x7f2d44100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.792+0000 7f2d4a7d0700 1 --2- 172.21.15.154:0/2991804108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d44100d20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f2d3800b5f0 tx=0x7f2d38006ee0).stop 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.792+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/2991804108 shutdown_connections 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.792+0000 7f2d4a7d0700 1 --2- 172.21.15.154:0/2991804108 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d44100d20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.792+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/2991804108 >> 172.21.15.154:0/2991804108 conn(0x7f2d440f63c0 msgr2=0x7f2d440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.792+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/2991804108 shutdown_connections 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.792+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/2991804108 wait complete. 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.793+0000 7f2d4a7d0700 1 Processor -- start 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.793+0000 7f2d4a7d0700 1 -- start start 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.793+0000 7f2d4a7d0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d4410cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.793+0000 7f2d4a7d0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4410d0b0 con 0x7f2d440703f0 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.793+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d4410cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.793+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d4410cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52408/0 (socket says 172.21.15.154:52408) 2024-01-16T17:10:47.777 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.793+0000 7f2d43fff700 1 -- 172.21.15.154:0/840287764 learned_addr learned my addr 172.21.15.154:0/840287764 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.793+0000 7f2d43fff700 1 -- 172.21.15.154:0/840287764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3800b200 con 0x7f2d440703f0 2024-01-16T17:10:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.794+0000 7f2d43fff700 1 --2- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d4410cbc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f2d3800b190 tx=0x7f2d38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.794+0000 7f2d40ff9700 1 -- 172.21.15.154:0/840287764 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d3801b410 con 0x7f2d440703f0 2024-01-16T17:10:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.794+0000 7f2d40ff9700 1 -- 172.21.15.154:0/840287764 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f2d3801b9e0 con 0x7f2d440703f0 2024-01-16T17:10:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.794+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4410d2b0 con 0x7f2d440703f0 2024-01-16T17:10:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.794+0000 7f2d40ff9700 1 -- 172.21.15.154:0/840287764 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d3801a880 con 0x7f2d440703f0 2024-01-16T17:10:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.794+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4410d6c0 con 0x7f2d440703f0 2024-01-16T17:10:47.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.795+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d44045ad0 con 0x7f2d440703f0 2024-01-16T17:10:47.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.795+0000 7f2d40ff9700 1 -- 172.21.15.154:0/840287764 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f2d3800f070 con 0x7f2d440703f0 2024-01-16T17:10:47.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.795+0000 7f2d40ff9700 1 --2- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f2d3002fe60 0x7f2d30032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:47.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.795+0000 7f2d40ff9700 1 -- 172.21.15.154:0/840287764 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2d38041ec0 con 0x7f2d440703f0 2024-01-16T17:10:47.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.795+0000 7f2d437fe700 1 --2- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f2d3002fe60 0x7f2d30032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:47.779 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.796+0000 7f2d437fe700 1 --2- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f2d3002fe60 0x7f2d30032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2d340096a0 tx=0x7f2d34006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:47.781 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.799+0000 7f2d40ff9700 1 -- 172.21.15.154:0/840287764 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d3802db70 con 0x7f2d440703f0 2024-01-16T17:10:47.957 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.975+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 --> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d4410dfa0 con 0x7f2d3002fe60 2024-01-16T17:10:47.961 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.978+0000 7f2d40ff9700 1 -- 172.21.15.154:0/840287764 <== mgr.14142 v2:172.21.15.154:6800/1485441924 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f2d4410dfa0 con 0x7f2d3002fe60 2024-01-16T17:10:47.961 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-16T17:10:47.963 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.980+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f2d3002fe60 msgr2=0x7f2d30032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:47.963 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.980+0000 7f2d4a7d0700 1 --2- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f2d3002fe60 0x7f2d30032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2d340096a0 tx=0x7f2d34006b40).stop 2024-01-16T17:10:47.963 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.980+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 msgr2=0x7f2d4410cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:47.963 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.980+0000 7f2d4a7d0700 1 --2- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d4410cbc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f2d3800b190 tx=0x7f2d38006ee0).stop 2024-01-16T17:10:47.964 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.981+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 shutdown_connections 2024-01-16T17:10:47.964 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.981+0000 7f2d4a7d0700 1 --2- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f2d3002fe60 0x7f2d30032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:47.964 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.981+0000 7f2d4a7d0700 1 --2- 172.21.15.154:0/840287764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d440703f0 0x7f2d4410cbc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:47.964 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.981+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 >> 172.21.15.154:0/840287764 conn(0x7f2d440f63c0 msgr2=0x7f2d440f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:47.964 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.981+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 shutdown_connections 2024-01-16T17:10:47.964 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:47.981+0000 7f2d4a7d0700 1 -- 172.21.15.154:0/840287764 wait complete. 2024-01-16T17:10:48.301 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-16T17:10:48.618 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.632+0000 7fee58637700 1 Processor -- start 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.632+0000 7fee58637700 1 -- start start 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.632+0000 7fee58637700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee50100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.632+0000 7fee58637700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee500fc750 con 0x7fee500703b0 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.632+0000 7fee563d3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee50100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.632+0000 7fee563d3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee50100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52420/0 (socket says 172.21.15.154:52420) 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.632+0000 7fee563d3700 1 -- 172.21.15.154:0/4214118405 learned_addr learned my addr 172.21.15.154:0/4214118405 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.633+0000 7fee563d3700 1 -- 172.21.15.154:0/4214118405 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50101250 con 0x7fee500703b0 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.633+0000 7fee563d3700 1 --2- 172.21.15.154:0/4214118405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee50100d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fee4c00b5f0 tx=0x7fee4c006ee0).ready entity=mon.0 client_cookie=19b0e2ca712f8a0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.633+0000 7fee553d1700 1 -- 172.21.15.154:0/4214118405 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4c00d600 con 0x7fee500703b0 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.633+0000 7fee553d1700 1 -- 172.21.15.154:0/4214118405 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fee4c00dbd0 con 0x7fee500703b0 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.633+0000 7fee553d1700 1 -- 172.21.15.154:0/4214118405 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4c011c00 con 0x7fee500703b0 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.633+0000 7fee58637700 1 -- 172.21.15.154:0/4214118405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 msgr2=0x7fee50100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.633+0000 7fee58637700 1 --2- 172.21.15.154:0/4214118405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee50100d60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fee4c00b5f0 tx=0x7fee4c006ee0).stop 2024-01-16T17:10:48.619 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.634+0000 7fee58637700 1 -- 172.21.15.154:0/4214118405 shutdown_connections 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.634+0000 7fee58637700 1 --2- 172.21.15.154:0/4214118405 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee50100d60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.634+0000 7fee58637700 1 -- 172.21.15.154:0/4214118405 >> 172.21.15.154:0/4214118405 conn(0x7fee500f63c0 msgr2=0x7fee500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.634+0000 7fee58637700 1 -- 172.21.15.154:0/4214118405 shutdown_connections 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.634+0000 7fee58637700 1 -- 172.21.15.154:0/4214118405 wait complete. 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.634+0000 7fee58637700 1 Processor -- start 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.634+0000 7fee58637700 1 -- start start 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee58637700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee5010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee58637700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee5010d1d0 con 0x7fee500703b0 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee563d3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee5010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee563d3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee5010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52432/0 (socket says 172.21.15.154:52432) 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee563d3700 1 -- 172.21.15.154:0/505412620 learned_addr learned my addr 172.21.15.154:0/505412620 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee563d3700 1 -- 172.21.15.154:0/505412620 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c00b200 con 0x7fee500703b0 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee563d3700 1 --2- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee5010cce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fee4c00e040 tx=0x7fee4c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee3effd700 1 -- 172.21.15.154:0/505412620 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4c00d9f0 con 0x7fee500703b0 2024-01-16T17:10:48.620 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee3effd700 1 -- 172.21.15.154:0/505412620 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fee4c01a9f0 con 0x7fee500703b0 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee3effd700 1 -- 172.21.15.154:0/505412620 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4c01b8c0 con 0x7fee500703b0 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5010d3d0 con 0x7fee500703b0 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.635+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee5010d7e0 con 0x7fee500703b0 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.636+0000 7fee3effd700 1 -- 172.21.15.154:0/505412620 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fee4c00f070 con 0x7fee500703b0 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.636+0000 7fee3effd700 1 --2- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fee3802fe60 0x7fee38032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.636+0000 7fee3effd700 1 -- 172.21.15.154:0/505412620 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fee4c042290 con 0x7fee500703b0 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.636+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee50045ad0 con 0x7fee500703b0 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.636+0000 7fee55bd2700 1 --2- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fee3802fe60 0x7fee38032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:48.621 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.637+0000 7fee55bd2700 1 --2- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fee3802fe60 0x7fee38032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fee400096a0 tx=0x7fee40006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:48.622 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.641+0000 7fee3effd700 1 -- 172.21.15.154:0/505412620 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee4c016070 con 0x7fee500703b0 2024-01-16T17:10:48.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:48 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:47.113333+0000 mgr.smithi154.sllivz (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:48.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:48 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:47.113982+0000 mgr.smithi154.sllivz (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-16T17:10:48.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:48 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:47.979835+0000 mon.smithi154 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.154:0/2663923502' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:48.772 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.790+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fee5010e680 con 0x7fee500703b0 2024-01-16T17:10:48.775 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.793+0000 7fee3effd700 1 -- 172.21.15.154:0/505412620 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fee4c022d20 con 0x7fee500703b0 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fee3802fe60 msgr2=0x7fee38032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 --2- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fee3802fe60 0x7fee38032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fee400096a0 tx=0x7fee40006b40).stop 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 msgr2=0x7fee5010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 --2- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee5010cce0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fee4c00e040 tx=0x7fee4c006ee0).stop 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 shutdown_connections 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 --2- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fee3802fe60 0x7fee38032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 --2- 172.21.15.154:0/505412620 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fee500703b0 0x7fee5010cce0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 >> 172.21.15.154:0/505412620 conn(0x7fee500f63c0 msgr2=0x7fee500f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 shutdown_connections 2024-01-16T17:10:48.778 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:48.795+0000 7fee58637700 1 -- 172.21.15.154:0/505412620 wait complete. 2024-01-16T17:10:49.118 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.466+0000 7f7dc59ee700 1 Processor -- start 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.466+0000 7f7dc59ee700 1 -- start start 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.466+0000 7f7dc59ee700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc00fc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.466+0000 7f7dc59ee700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc00fcbd0 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.467+0000 7f7dbeffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc00fc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.467+0000 7f7dbeffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc00fc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52448/0 (socket says 172.21.15.154:52448) 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.467+0000 7f7dbeffd700 1 -- 172.21.15.154:0/101911400 learned_addr learned my addr 172.21.15.154:0/101911400 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.467+0000 7f7dbeffd700 1 -- 172.21.15.154:0/101911400 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc00fcd10 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.467+0000 7f7dbeffd700 1 --2- 172.21.15.154:0/101911400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc00fc6e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7db400b5f0 tx=0x7f7db4006ee0).ready entity=mon.0 client_cookie=9d12e67f936c8daa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.467+0000 7f7dbdffb700 1 -- 172.21.15.154:0/101911400 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7db400d600 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.467+0000 7f7dbdffb700 1 -- 172.21.15.154:0/101911400 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7db400dbd0 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.467+0000 7f7dbdffb700 1 -- 172.21.15.154:0/101911400 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7db4011c00 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.451 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.468+0000 7f7dc59ee700 1 -- 172.21.15.154:0/101911400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 msgr2=0x7f7dc00fc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:49.452 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.468+0000 7f7dc59ee700 1 --2- 172.21.15.154:0/101911400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc00fc6e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7db400b5f0 tx=0x7f7db4006ee0).stop 2024-01-16T17:10:49.452 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.468+0000 7f7dc59ee700 1 -- 172.21.15.154:0/101911400 shutdown_connections 2024-01-16T17:10:49.452 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.468+0000 7f7dc59ee700 1 --2- 172.21.15.154:0/101911400 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc00fc6e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:49.452 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.468+0000 7f7dc59ee700 1 -- 172.21.15.154:0/101911400 >> 172.21.15.154:0/101911400 conn(0x7f7dc00f5b10 msgr2=0x7f7dc00f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:49.452 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.468+0000 7f7dc59ee700 1 -- 172.21.15.154:0/101911400 shutdown_connections 2024-01-16T17:10:49.452 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.468+0000 7f7dc59ee700 1 -- 172.21.15.154:0/101911400 wait complete. 2024-01-16T17:10:49.452 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.469+0000 7f7dc59ee700 1 Processor -- start 2024-01-16T17:10:49.452 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.469+0000 7f7dc59ee700 1 -- start start 2024-01-16T17:10:49.454 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.469+0000 7f7dc59ee700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc010cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:49.454 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.469+0000 7f7dc59ee700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc010d0b0 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.454 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.469+0000 7f7dbeffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc010cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:49.454 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.469+0000 7f7dbeffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc010cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52462/0 (socket says 172.21.15.154:52462) 2024-01-16T17:10:49.454 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.469+0000 7f7dbeffd700 1 -- 172.21.15.154:0/1452329635 learned_addr learned my addr 172.21.15.154:0/1452329635 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:49.454 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.469+0000 7f7dbeffd700 1 -- 172.21.15.154:0/1452329635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db400b200 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.470+0000 7f7dbeffd700 1 --2- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc010cbc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7db400b1d0 tx=0x7f7db4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.470+0000 7f7da77fe700 1 -- 172.21.15.154:0/1452329635 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7db401b410 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.470+0000 7f7da77fe700 1 -- 172.21.15.154:0/1452329635 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7db401b9e0 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.470+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc010d2b0 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.470+0000 7f7da77fe700 1 -- 172.21.15.154:0/1452329635 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7db401a840 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.470+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc010d6c0 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.470+0000 7f7da77fe700 1 -- 172.21.15.154:0/1452329635 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7db400f070 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.471+0000 7f7da77fe700 1 --2- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f7dac02fe60 0x7f7dac032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.471+0000 7f7da77fe700 1 -- 172.21.15.154:0/1452329635 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7db40413f0 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.471+0000 7f7dbe7fc700 1 --2- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f7dac02fe60 0x7f7dac032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.471+0000 7f7dbe7fc700 1 --2- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f7dac02fe60 0x7f7dac032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7db00096a0 tx=0x7f7db0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:49.455 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.471+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc0059100 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.457 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.475+0000 7f7da77fe700 1 -- 172.21.15.154:0/1452329635 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7db4025460 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:47.977081+0000 mgr.smithi154.sllivz (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:49.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:10:47.977628+0000 mgr.smithi154.sllivz (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-16T17:10:49.460 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:48.794316+0000 mon.smithi154 (mon.0) 81 : audit [INF] from='client.? 172.21.15.154:0/505412620' entity='client.admin' 2024-01-16T17:10:49.607 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.624+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f7dc0045ad0 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.609 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.627+0000 7f7da77fe700 1 -- 172.21.15.154:0/1452329635 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7db4022310 con 0x7f7dc00fc2c0 2024-01-16T17:10:49.612 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f7dac02fe60 msgr2=0x7f7dac032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:49.612 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 --2- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f7dac02fe60 0x7f7dac032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7db00096a0 tx=0x7f7db0006b40).stop 2024-01-16T17:10:49.612 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 msgr2=0x7f7dc010cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:49.612 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 --2- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc010cbc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7db400b1d0 tx=0x7f7db4006ee0).stop 2024-01-16T17:10:49.612 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 shutdown_connections 2024-01-16T17:10:49.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 --2- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f7dac02fe60 0x7f7dac032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:49.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 --2- 172.21.15.154:0/1452329635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7dc00fc2c0 0x7f7dc010cbc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:49.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 >> 172.21.15.154:0/1452329635 conn(0x7f7dc00f5b10 msgr2=0x7f7dc00f7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:49.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 shutdown_connections 2024-01-16T17:10:49.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:49.629+0000 7f7dc59ee700 1 -- 172.21.15.154:0/1452329635 wait complete. 2024-01-16T17:10:49.951 INFO:teuthology.orchestra.run.smithi154.stderr:Enabling the dashboard module... 2024-01-16T17:10:49.951 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-16T17:10:50.291 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.307+0000 7f47245b5700 1 Processor -- start 2024-01-16T17:10:50.291 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.307+0000 7f47245b5700 1 -- start start 2024-01-16T17:10:50.291 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.307+0000 7f47245b5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:50.291 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.307+0000 7f47245b5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0fee10 con 0x7f471c0703f0 2024-01-16T17:10:50.291 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.308+0000 7f4722351700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:50.291 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.308+0000 7f4722351700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52478/0 (socket says 172.21.15.154:52478) 2024-01-16T17:10:50.291 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.308+0000 7f4722351700 1 -- 172.21.15.154:0/2360137582 learned_addr learned my addr 172.21.15.154:0/2360137582 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:50.291 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.308+0000 7f4722351700 1 -- 172.21.15.154:0/2360137582 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c0fef50 con 0x7f471c0703f0 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.308+0000 7f4722351700 1 --2- 172.21.15.154:0/2360137582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c0fe920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f471800b5f0 tx=0x7f4718006ee0).ready entity=mon.0 client_cookie=e72097c0fd176001 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.308+0000 7f472134f700 1 -- 172.21.15.154:0/2360137582 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f471800d600 con 0x7f471c0703f0 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.308+0000 7f472134f700 1 -- 172.21.15.154:0/2360137582 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f471800dbd0 con 0x7f471c0703f0 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.308+0000 7f472134f700 1 -- 172.21.15.154:0/2360137582 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4718011c00 con 0x7f471c0703f0 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.309+0000 7f47245b5700 1 -- 172.21.15.154:0/2360137582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 msgr2=0x7f471c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.309+0000 7f47245b5700 1 --2- 172.21.15.154:0/2360137582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c0fe920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f471800b5f0 tx=0x7f4718006ee0).stop 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.309+0000 7f47245b5700 1 -- 172.21.15.154:0/2360137582 shutdown_connections 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.309+0000 7f47245b5700 1 --2- 172.21.15.154:0/2360137582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c0fe920 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.309+0000 7f47245b5700 1 -- 172.21.15.154:0/2360137582 >> 172.21.15.154:0/2360137582 conn(0x7f471c0f63c0 msgr2=0x7f471c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.309+0000 7f47245b5700 1 -- 172.21.15.154:0/2360137582 shutdown_connections 2024-01-16T17:10:50.292 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.309+0000 7f47245b5700 1 -- 172.21.15.154:0/2360137582 wait complete. 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.309+0000 7f47245b5700 1 Processor -- start 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f47245b5700 1 -- start start 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f47245b5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c10a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f47245b5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c10aeb0 con 0x7f471c0703f0 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f4722351700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c10a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f4722351700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c10a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52488/0 (socket says 172.21.15.154:52488) 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f4722351700 1 -- 172.21.15.154:0/1155528139 learned_addr learned my addr 172.21.15.154:0/1155528139 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f4722351700 1 -- 172.21.15.154:0/1155528139 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471800b200 con 0x7f471c0703f0 2024-01-16T17:10:50.294 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f4722351700 1 --2- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c10a9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f471800b190 tx=0x7f4718006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f470affd700 1 -- 172.21.15.154:0/1155528139 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f471801b410 con 0x7f471c0703f0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f470affd700 1 -- 172.21.15.154:0/1155528139 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f471801b9e0 con 0x7f471c0703f0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c10b0b0 con 0x7f471c0703f0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.310+0000 7f470affd700 1 -- 172.21.15.154:0/1155528139 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f471801a880 con 0x7f471c0703f0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.311+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c10b4c0 con 0x7f471c0703f0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.311+0000 7f470affd700 1 -- 172.21.15.154:0/1155528139 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f471800f070 con 0x7f471c0703f0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.311+0000 7f470affd700 1 --2- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f470402fe60 0x7f4704032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.311+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c045ad0 con 0x7f471c0703f0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.312+0000 7f470affd700 1 -- 172.21.15.154:0/1155528139 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4718041410 con 0x7f471c0703f0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.312+0000 7f4721b50700 1 --2- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f470402fe60 0x7f4704032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:50.295 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.312+0000 7f4721b50700 1 --2- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f470402fe60 0x7f4704032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f470c0096a0 tx=0x7f470c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:50.298 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.316+0000 7f470affd700 1 -- 172.21.15.154:0/1155528139 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4718019cd0 con 0x7f471c0703f0 2024-01-16T17:10:50.484 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.502+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f471c0ffbd0 con 0x7f471c0703f0 2024-01-16T17:10:50.609 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.627+0000 7f470affd700 1 -- 172.21.15.154:0/1155528139 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f471801bb50 con 0x7f471c0703f0 2024-01-16T17:10:50.610 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.629+0000 7f470affd700 1 -- 172.21.15.154:0/1155528139 <== mon.0 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7f47180058e0 con 0x7f471c0703f0 2024-01-16T17:10:50.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.630+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f470402fe60 msgr2=0x7f4704032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:50.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.630+0000 7f47245b5700 1 --2- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f470402fe60 0x7f4704032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f470c0096a0 tx=0x7f470c006b40).stop 2024-01-16T17:10:50.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.630+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 msgr2=0x7f471c10a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:50.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.630+0000 7f47245b5700 1 --2- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c10a9c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f471800b190 tx=0x7f4718006ee0).stop 2024-01-16T17:10:50.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.630+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 shutdown_connections 2024-01-16T17:10:50.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.630+0000 7f47245b5700 1 --2- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f470402fe60 0x7f4704032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:50.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.630+0000 7f47245b5700 1 --2- 172.21.15.154:0/1155528139 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f471c0703f0 0x7f471c10a9c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:50.613 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.630+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 >> 172.21.15.154:0/1155528139 conn(0x7f471c0f63c0 msgr2=0x7f471c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:50.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.632+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 shutdown_connections 2024-01-16T17:10:50.614 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:50.632+0000 7f47245b5700 1 -- 172.21.15.154:0/1155528139 wait complete. 2024-01-16T17:10:50.895 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:50 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:49.628955+0000 mon.smithi154 (mon.0) 82 : audit [INF] from='client.? 172.21.15.154:0/1452329635' entity='client.admin' 2024-01-16T17:10:50.895 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:50 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:50.504006+0000 mon.smithi154 (mon.0) 83 : audit [INF] from='client.? 172.21.15.154:0/1155528139' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-16T17:10:50.951 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-16T17:10:51.273 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.289+0000 7f17c94d0700 1 Processor -- start 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.289+0000 7f17c94d0700 1 -- start start 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.289+0000 7f17c94d0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c40fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.289+0000 7f17c94d0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c40fec60 con 0x7f17c40703f0 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.289+0000 7f17c3fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c40fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.289+0000 7f17c3fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c40fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52504/0 (socket says 172.21.15.154:52504) 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.289+0000 7f17c3fff700 1 -- 172.21.15.154:0/3468051397 learned_addr learned my addr 172.21.15.154:0/3468051397 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.289+0000 7f17c3fff700 1 -- 172.21.15.154:0/3468051397 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c40feda0 con 0x7f17c40703f0 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c3fff700 1 --2- 172.21.15.154:0/3468051397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c40fe770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f17b400b5f0 tx=0x7f17b4006ee0).ready entity=mon.0 client_cookie=f9c0c422561a2bcb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c2ffd700 1 -- 172.21.15.154:0/3468051397 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17b400d600 con 0x7f17c40703f0 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c2ffd700 1 -- 172.21.15.154:0/3468051397 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f17b400dbd0 con 0x7f17c40703f0 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c2ffd700 1 -- 172.21.15.154:0/3468051397 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17b4011c00 con 0x7f17c40703f0 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c94d0700 1 -- 172.21.15.154:0/3468051397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 msgr2=0x7f17c40fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c94d0700 1 --2- 172.21.15.154:0/3468051397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c40fe770 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f17b400b5f0 tx=0x7f17b4006ee0).stop 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c94d0700 1 -- 172.21.15.154:0/3468051397 shutdown_connections 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c94d0700 1 --2- 172.21.15.154:0/3468051397 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c40fe770 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:51.274 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c94d0700 1 -- 172.21.15.154:0/3468051397 >> 172.21.15.154:0/3468051397 conn(0x7f17c40f6210 msgr2=0x7f17c40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c94d0700 1 -- 172.21.15.154:0/3468051397 shutdown_connections 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.290+0000 7f17c94d0700 1 -- 172.21.15.154:0/3468051397 wait complete. 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.291+0000 7f17c94d0700 1 Processor -- start 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.291+0000 7f17c94d0700 1 -- start start 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.291+0000 7f17c94d0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c410a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.291+0000 7f17c94d0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c410ad00 con 0x7f17c40703f0 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.291+0000 7f17c3fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c410a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.291+0000 7f17c3fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c410a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52506/0 (socket says 172.21.15.154:52506) 2024-01-16T17:10:51.275 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.291+0000 7f17c3fff700 1 -- 172.21.15.154:0/3250283725 learned_addr learned my addr 172.21.15.154:0/3250283725 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:51.276 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.292+0000 7f17c3fff700 1 -- 172.21.15.154:0/3250283725 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b400b200 con 0x7f17c40703f0 2024-01-16T17:10:51.276 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.292+0000 7f17c3fff700 1 --2- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c410a810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f17b400b190 tx=0x7f17b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:51.276 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.292+0000 7f17c0ff9700 1 -- 172.21.15.154:0/3250283725 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17b401b410 con 0x7f17c40703f0 2024-01-16T17:10:51.276 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.292+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c410af00 con 0x7f17c40703f0 2024-01-16T17:10:51.276 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.292+0000 7f17c0ff9700 1 -- 172.21.15.154:0/3250283725 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f17b401b9e0 con 0x7f17c40703f0 2024-01-16T17:10:51.276 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.292+0000 7f17c0ff9700 1 -- 172.21.15.154:0/3250283725 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17b401a880 con 0x7f17c40703f0 2024-01-16T17:10:51.276 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.292+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c410b310 con 0x7f17c40703f0 2024-01-16T17:10:51.276 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.293+0000 7f17c0ff9700 1 -- 172.21.15.154:0/3250283725 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f17b400f070 con 0x7f17c40703f0 2024-01-16T17:10:51.277 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.293+0000 7f17c0ff9700 1 --2- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f17b002feb0 0x7f17b0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:51.277 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.293+0000 7f17c0ff9700 1 -- 172.21.15.154:0/3250283725 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f17b4041520 con 0x7f17c40703f0 2024-01-16T17:10:51.277 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.293+0000 7f17c37fe700 1 -- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f17b002feb0 msgr2=0x7f17b0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/1485441924 2024-01-16T17:10:51.277 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.293+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c4045ad0 con 0x7f17c40703f0 2024-01-16T17:10:51.277 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.293+0000 7f17c37fe700 1 --2- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f17b002feb0 0x7f17b0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-16T17:10:51.279 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.297+0000 7f17c0ff9700 1 -- 172.21.15.154:0/3250283725 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17b4016070 con 0x7f17c40703f0 2024-01-16T17:10:51.473 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.491+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f17c4104640 con 0x7f17c40703f0 2024-01-16T17:10:51.473 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.491+0000 7f17c0ff9700 1 -- 172.21.15.154:0/3250283725 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7f17b4022b90 con 0x7f17c40703f0 2024-01-16T17:10:51.474 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: { 2024-01-16T17:10:51.474 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "epoch": 12, 2024-01-16T17:10:51.474 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "available": true, 2024-01-16T17:10:51.474 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "active_name": "smithi154.sllivz", 2024-01-16T17:10:51.474 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-16T17:10:51.474 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: } 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f17b002feb0 msgr2=0x7f17b0032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 --2- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f17b002feb0 0x7f17b0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 msgr2=0x7f17c410a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 --2- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c410a810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f17b400b190 tx=0x7f17b4006ee0).stop 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 shutdown_connections 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 --2- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7f17b002feb0 0x7f17b0032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 --2- 172.21.15.154:0/3250283725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f17c40703f0 0x7f17c410a810 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 >> 172.21.15.154:0/3250283725 conn(0x7f17c40f6210 msgr2=0x7f17c40f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:51.476 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 shutdown_connections 2024-01-16T17:10:51.477 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:51.493+0000 7f17c94d0700 1 -- 172.21.15.154:0/3250283725 wait complete. 2024-01-16T17:10:51.778 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:50.630117+0000 mon.smithi154 (mon.0) 84 : audit [INF] from='client.? 172.21.15.154:0/1155528139' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-16T17:10:51.778 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:50.630163+0000 mon.smithi154 (mon.0) 85 : cluster [DBG] mgrmap e12: smithi154.sllivz(active, since 6s) 2024-01-16T17:10:51.778 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:51.492698+0000 mon.smithi154 (mon.0) 86 : audit [DBG] from='client.? 172.21.15.154:0/3250283725' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-16T17:10:51.834 INFO:teuthology.orchestra.run.smithi154.stderr:Waiting for the mgr to restart... 2024-01-16T17:10:51.835 INFO:teuthology.orchestra.run.smithi154.stderr:Waiting for mgr epoch 12... 2024-01-16T17:10:51.835 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-16T17:10:52.349 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.365+0000 7fbe96868700 1 Processor -- start 2024-01-16T17:10:52.349 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.365+0000 7fbe96868700 1 -- start start 2024-01-16T17:10:52.349 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.365+0000 7fbe96868700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe900fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:52.349 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.365+0000 7fbe96868700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe900fb490 con 0x7fbe900fab80 2024-01-16T17:10:52.349 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.365+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe900fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:52.349 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.365+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe900fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52518/0 (socket says 172.21.15.154:52518) 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.365+0000 7fbe8ffff700 1 -- 172.21.15.154:0/1269639437 learned_addr learned my addr 172.21.15.154:0/1269639437 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.366+0000 7fbe8ffff700 1 -- 172.21.15.154:0/1269639437 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe900fb5d0 con 0x7fbe900fab80 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.366+0000 7fbe8ffff700 1 --2- 172.21.15.154:0/1269639437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe900fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbe8000b870 tx=0x7fbe80006f70).ready entity=mon.0 client_cookie=b110b7ddbcee82fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.366+0000 7fbe8effd700 1 -- 172.21.15.154:0/1269639437 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe8000da20 con 0x7fbe900fab80 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.366+0000 7fbe8effd700 1 -- 172.21.15.154:0/1269639437 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbe80010470 con 0x7fbe900fab80 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.366+0000 7fbe8effd700 1 -- 172.21.15.154:0/1269639437 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe8000f5b0 con 0x7fbe900fab80 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.367+0000 7fbe96868700 1 -- 172.21.15.154:0/1269639437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 msgr2=0x7fbe900fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.367+0000 7fbe96868700 1 --2- 172.21.15.154:0/1269639437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe900fafa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbe8000b870 tx=0x7fbe80006f70).stop 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.367+0000 7fbe96868700 1 -- 172.21.15.154:0/1269639437 shutdown_connections 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.367+0000 7fbe96868700 1 --2- 172.21.15.154:0/1269639437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe900fafa0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.367+0000 7fbe96868700 1 -- 172.21.15.154:0/1269639437 >> 172.21.15.154:0/1269639437 conn(0x7fbe900f63c0 msgr2=0x7fbe900f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.367+0000 7fbe96868700 1 -- 172.21.15.154:0/1269639437 shutdown_connections 2024-01-16T17:10:52.350 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.367+0000 7fbe96868700 1 -- 172.21.15.154:0/1269639437 wait complete. 2024-01-16T17:10:52.352 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.368+0000 7fbe96868700 1 Processor -- start 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.368+0000 7fbe96868700 1 -- start start 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.368+0000 7fbe96868700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe90108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.368+0000 7fbe96868700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe90108c60 con 0x7fbe900fab80 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.368+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe90108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.368+0000 7fbe8ffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe90108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:52524/0 (socket says 172.21.15.154:52524) 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.368+0000 7fbe8ffff700 1 -- 172.21.15.154:0/3434018832 learned_addr learned my addr 172.21.15.154:0/3434018832 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.369+0000 7fbe8ffff700 1 -- 172.21.15.154:0/3434018832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8000b480 con 0x7fbe900fab80 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.369+0000 7fbe8ffff700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe90108770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbe8000de70 tx=0x7fbe80006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.369+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe8000f900 con 0x7fbe900fab80 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.369+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe90108e60 con 0x7fbe900fab80 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.369+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbe8000fed0 con 0x7fbe900fab80 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.369+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe80019e80 con 0x7fbe900fab80 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.369+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe90109270 con 0x7fbe900fab80 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.370+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbe8002a070 con 0x7fbe900fab80 2024-01-16T17:10:52.353 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.370+0000 7fbe8cff9700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 0x7fbe7c0367c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:52.354 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.370+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 --> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe7c036e80 con 0x7fbe7c034300 2024-01-16T17:10:52.354 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.370+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbe80041c00 con 0x7fbe900fab80 2024-01-16T17:10:52.354 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.370+0000 7fbe8f7fe700 1 -- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 msgr2=0x7fbe7c0367c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/1485441924 2024-01-16T17:10:52.354 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.370+0000 7fbe8f7fe700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 0x7fbe7c0367c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-16T17:10:52.553 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.571+0000 7fbe8f7fe700 1 -- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 msgr2=0x7fbe7c0367c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/1485441924 2024-01-16T17:10:52.553 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.571+0000 7fbe8f7fe700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 0x7fbe7c0367c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-16T17:10:52.954 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.972+0000 7fbe8f7fe700 1 -- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 msgr2=0x7fbe7c0367c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/1485441924 2024-01-16T17:10:52.954 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:52.972+0000 7fbe8f7fe700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 0x7fbe7c0367c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-16T17:10:53.755 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:53.773+0000 7fbe8f7fe700 1 -- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 msgr2=0x7fbe7c0367c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/1485441924 2024-01-16T17:10:53.755 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:53.773+0000 7fbe8f7fe700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 0x7fbe7c0367c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-16T17:10:55.357 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:55.374+0000 7fbe8f7fe700 1 -- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 msgr2=0x7fbe7c0367c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:6800/1485441924 2024-01-16T17:10:55.357 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:55.374+0000 7fbe8f7fe700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 0x7fbe7c0367c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-16T17:10:55.762 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:55.779+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fbe80016030 con 0x7fbe900fab80 2024-01-16T17:10:55.762 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:55.779+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 msgr2=0x7fbe7c0367c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T17:10:55.762 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:55.780+0000 7fbe8cff9700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 0x7fbe7c0367c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:56.765 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.781+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fbe8000fc30 con 0x7fbe900fab80 2024-01-16T17:10:56.765 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.781+0000 7fbe8cff9700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbe7c037f90 0x7fbe7c03a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:56.765 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.781+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 --> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe7c036e80 con 0x7fbe7c037f90 2024-01-16T17:10:56.765 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.781+0000 7fbe8f7fe700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbe7c037f90 0x7fbe7c03a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:56.765 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.782+0000 7fbe8f7fe700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbe7c037f90 0x7fbe7c03a380 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fbe84006fd0 tx=0x7fbe84006cf0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:56.765 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.782+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mgr.14162 v2:172.21.15.154:6800/2708438664 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fbe7c036e80 con 0x7fbe7c037f90 2024-01-16T17:10:56.770 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.788+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 --> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbe90059100 con 0x7fbe7c037f90 2024-01-16T17:10:56.770 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe8cff9700 1 -- 172.21.15.154:0/3434018832 <== mgr.14162 v2:172.21.15.154:6800/2708438664 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fbe90059100 con 0x7fbe7c037f90 2024-01-16T17:10:56.772 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: { 2024-01-16T17:10:56.772 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2024-01-16T17:10:56.772 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: "initialized": true 2024-01-16T17:10:56.772 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: } 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbe7c037f90 msgr2=0x7fbe7c03a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbe7c037f90 0x7fbe7c03a380 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fbe84006fd0 tx=0x7fbe84006cf0).stop 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 msgr2=0x7fbe90108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe90108770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbe8000de70 tx=0x7fbe80006f70).stop 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 shutdown_connections 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbe7c037f90 0x7fbe7c03a380 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe900fab80 0x7fbe90108770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 --2- 172.21.15.154:0/3434018832 >> [v2:172.21.15.154:6800/1485441924,v1:172.21.15.154:6801/1485441924] conn(0x7fbe7c034300 0x7fbe7c0367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 >> 172.21.15.154:0/3434018832 conn(0x7fbe900f63c0 msgr2=0x7fbe900ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 shutdown_connections 2024-01-16T17:10:56.773 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:56.789+0000 7fbe96868700 1 -- 172.21.15.154:0/3434018832 wait complete. 2024-01-16T17:10:56.965 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:55.729041+0000 mon.smithi154 (mon.0) 87 : cluster [INF] Active manager daemon smithi154.sllivz restarted 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:55.729568+0000 mon.smithi154 (mon.0) 88 : cluster [INF] Activating manager daemon smithi154.sllivz 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:55.731155+0000 mon.smithi154 (mon.0) 89 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:55.781831+0000 mon.smithi154 (mon.0) 90 : cluster [DBG] mgrmap e13: smithi154.sllivz(active, starting, since 0.0523843s) 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.782801+0000 mon.smithi154 (mon.0) 91 : audit [DBG] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mon metadata", "id": "smithi154"}]: dispatch 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.783327+0000 mon.smithi154 (mon.0) 92 : audit [DBG] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mgr metadata", "who": "smithi154.sllivz", "id": "smithi154.sllivz"}]: dispatch 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.783968+0000 mon.smithi154 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.784110+0000 mon.smithi154 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.784228+0000 mon.smithi154 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:55.797540+0000 mon.smithi154 (mon.0) 96 : cluster [INF] Manager daemon smithi154.sllivz is now available 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.897878+0000 mon.smithi154 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.898989+0000 mon.smithi154 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.923200+0000 mon.smithi154 (mon.0) 99 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:10:56.966 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:55.925883+0000 mon.smithi154 (mon.0) 100 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:10:57.125 INFO:teuthology.orchestra.run.smithi154.stderr:mgr epoch 12 is available 2024-01-16T17:10:57.125 INFO:teuthology.orchestra.run.smithi154.stderr:Generating a dashboard self-signed certificate... 2024-01-16T17:10:57.125 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-16T17:10:57.480 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.498+0000 7f6f4a79a700 1 Processor -- start 2024-01-16T17:10:57.482 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.498+0000 7f6f4a79a700 1 -- start start 2024-01-16T17:10:57.482 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.498+0000 7f6f4a79a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f440fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:57.482 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.498+0000 7f6f4a79a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f440fee10 con 0x7f6f440703f0 2024-01-16T17:10:57.482 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.498+0000 7f6f43fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f440fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:57.482 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.498+0000 7f6f43fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f440fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47564/0 (socket says 172.21.15.154:47564) 2024-01-16T17:10:57.482 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.498+0000 7f6f43fff700 1 -- 172.21.15.154:0/3928461259 learned_addr learned my addr 172.21.15.154:0/3928461259 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.499+0000 7f6f43fff700 1 -- 172.21.15.154:0/3928461259 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f440fef50 con 0x7f6f440703f0 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.499+0000 7f6f43fff700 1 --2- 172.21.15.154:0/3928461259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f440fe920 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6f3800b5f0 tx=0x7f6f38006ee0).ready entity=mon.0 client_cookie=bc9914c1d9e214b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.499+0000 7f6f42ffd700 1 -- 172.21.15.154:0/3928461259 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f3800d600 con 0x7f6f440703f0 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.499+0000 7f6f42ffd700 1 -- 172.21.15.154:0/3928461259 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6f3800dbd0 con 0x7f6f440703f0 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.499+0000 7f6f42ffd700 1 -- 172.21.15.154:0/3928461259 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f38011c00 con 0x7f6f440703f0 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.499+0000 7f6f4a79a700 1 -- 172.21.15.154:0/3928461259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 msgr2=0x7f6f440fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.499+0000 7f6f4a79a700 1 --2- 172.21.15.154:0/3928461259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f440fe920 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6f3800b5f0 tx=0x7f6f38006ee0).stop 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.500+0000 7f6f4a79a700 1 -- 172.21.15.154:0/3928461259 shutdown_connections 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.500+0000 7f6f4a79a700 1 --2- 172.21.15.154:0/3928461259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f440fe920 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.500+0000 7f6f4a79a700 1 -- 172.21.15.154:0/3928461259 >> 172.21.15.154:0/3928461259 conn(0x7f6f440f63c0 msgr2=0x7f6f440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.500+0000 7f6f4a79a700 1 -- 172.21.15.154:0/3928461259 shutdown_connections 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.500+0000 7f6f4a79a700 1 -- 172.21.15.154:0/3928461259 wait complete. 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.500+0000 7f6f4a79a700 1 Processor -- start 2024-01-16T17:10:57.483 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f4a79a700 1 -- start start 2024-01-16T17:10:57.485 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f4a79a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f4410aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:57.485 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f4a79a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4410af40 con 0x7f6f440703f0 2024-01-16T17:10:57.485 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f43fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f4410aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:57.485 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f43fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f4410aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47566/0 (socket says 172.21.15.154:47566) 2024-01-16T17:10:57.485 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f43fff700 1 -- 172.21.15.154:0/103430599 learned_addr learned my addr 172.21.15.154:0/103430599 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:57.485 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f43fff700 1 -- 172.21.15.154:0/103430599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3800b200 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f43fff700 1 --2- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f4410aa50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6f3800e040 tx=0x7f6f38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f40ff9700 1 -- 172.21.15.154:0/103430599 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f3800d9f0 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4410b140 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4410b550 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.501+0000 7f6f40ff9700 1 -- 172.21.15.154:0/103430599 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6f3801a9f0 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.502+0000 7f6f40ff9700 1 -- 172.21.15.154:0/103430599 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f380258a0 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.502+0000 7f6f40ff9700 1 -- 172.21.15.154:0/103430599 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f6f3800f070 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.502+0000 7f6f40ff9700 1 --2- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f3002fd70 0x7f6f30032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.502+0000 7f6f40ff9700 1 -- 172.21.15.154:0/103430599 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f380413a0 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.503+0000 7f6f437fe700 1 --2- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f3002fd70 0x7f6f30032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.503+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f24004fa0 con 0x7f6f440703f0 2024-01-16T17:10:57.486 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.503+0000 7f6f437fe700 1 --2- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f3002fd70 0x7f6f30032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6f340096a0 tx=0x7f6f34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:57.489 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.507+0000 7f6f40ff9700 1 -- 172.21.15.154:0/103430599 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f38016080 con 0x7f6f440703f0 2024-01-16T17:10:57.642 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.660+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 --> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f24002a50 con 0x7f6f3002fd70 2024-01-16T17:10:57.757 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.775+0000 7f6f40ff9700 1 -- 172.21.15.154:0/103430599 <== mgr.14162 v2:172.21.15.154:6800/2708438664 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f6f24002a50 con 0x7f6f3002fd70 2024-01-16T17:10:57.757 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-16T17:10:57.759 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.776+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f3002fd70 msgr2=0x7f6f30032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:57.759 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.776+0000 7f6f4a79a700 1 --2- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f3002fd70 0x7f6f30032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f6f340096a0 tx=0x7f6f34006b40).stop 2024-01-16T17:10:57.759 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.776+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 msgr2=0x7f6f4410aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:57.759 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.776+0000 7f6f4a79a700 1 --2- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f4410aa50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f6f3800e040 tx=0x7f6f38006ee0).stop 2024-01-16T17:10:57.759 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.776+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 shutdown_connections 2024-01-16T17:10:57.759 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.776+0000 7f6f4a79a700 1 --2- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f3002fd70 0x7f6f30032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:57.759 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.776+0000 7f6f4a79a700 1 --2- 172.21.15.154:0/103430599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f440703f0 0x7f6f4410aa50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:57.759 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.776+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 >> 172.21.15.154:0/103430599 conn(0x7f6f440f63c0 msgr2=0x7f6f440f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:57.760 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.777+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 shutdown_connections 2024-01-16T17:10:57.760 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:57.777+0000 7f6f4a79a700 1 -- 172.21.15.154:0/103430599 wait complete. 2024-01-16T17:10:58.035 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:56.783722+0000 mgr.smithi154.sllivz (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-16T17:10:58.035 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:56.784346+0000 mon.smithi154 (mon.0) 101 : cluster [DBG] mgrmap e14: smithi154.sllivz(active, since 1.0549s) 2024-01-16T17:10:58.035 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:56.790256+0000 mgr.smithi154.sllivz (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-16T17:10:58.035 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:57.662490+0000 mgr.smithi154.sllivz (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:58.035 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:57.774249+0000 mon.smithi154 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:58.036 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:57.776027+0000 mon.smithi154 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:58.064 INFO:teuthology.orchestra.run.smithi154.stderr:Creating initial admin user... 2024-01-16T17:10:58.064 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmprd6w4udv:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.417+0000 7f4a9cf47700 1 Processor -- start 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.417+0000 7f4a9cf47700 1 -- start start 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.417+0000 7f4a9cf47700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a98070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.417+0000 7f4a9cf47700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980728d0 con 0x7f4a980724b0 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.418+0000 7f4a9659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a98070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.418+0000 7f4a9659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a98070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47578/0 (socket says 172.21.15.154:47578) 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.418+0000 7f4a9659c700 1 -- 172.21.15.154:0/1821692468 learned_addr learned my addr 172.21.15.154:0/1821692468 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.418+0000 7f4a9659c700 1 -- 172.21.15.154:0/1821692468 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a98070f40 con 0x7f4a980724b0 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.418+0000 7f4a9659c700 1 --2- 172.21.15.154:0/1821692468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a98070a50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4a90009960 tx=0x7f4a900092d0).ready entity=mon.0 client_cookie=6b211244e3ea4d14 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.418+0000 7f4a9559a700 1 -- 172.21.15.154:0/1821692468 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a9000b9c0 con 0x7f4a980724b0 2024-01-16T17:10:58.402 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.418+0000 7f4a9559a700 1 -- 172.21.15.154:0/1821692468 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a90011400 con 0x7f4a980724b0 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.418+0000 7f4a9559a700 1 -- 172.21.15.154:0/1821692468 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a900103a0 con 0x7f4a980724b0 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1821692468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 msgr2=0x7f4a98070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 --2- 172.21.15.154:0/1821692468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a98070a50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f4a90009960 tx=0x7f4a900092d0).stop 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1821692468 shutdown_connections 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 --2- 172.21.15.154:0/1821692468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a98070a50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1821692468 >> 172.21.15.154:0/1821692468 conn(0x7f4a980f74c0 msgr2=0x7f4a980f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1821692468 shutdown_connections 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1821692468 wait complete. 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 Processor -- start 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.419+0000 7f4a9cf47700 1 -- start start 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9cf47700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a9810cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9cf47700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9810d1f0 con 0x7f4a980724b0 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a9810cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a9810cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47580/0 (socket says 172.21.15.154:47580) 2024-01-16T17:10:58.403 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9659c700 1 -- 172.21.15.154:0/1481276940 learned_addr learned my addr 172.21.15.154:0/1481276940 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:58.404 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9659c700 1 -- 172.21.15.154:0/1481276940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a90009570 con 0x7f4a980724b0 2024-01-16T17:10:58.404 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9659c700 1 --2- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a9810cd00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f4a90000f80 tx=0x7f4a900092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:58.404 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a86ffd700 1 -- 172.21.15.154:0/1481276940 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a90010820 con 0x7f4a980724b0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a86ffd700 1 -- 172.21.15.154:0/1481276940 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4a90010df0 con 0x7f4a980724b0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9810d3f0 con 0x7f4a980724b0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a86ffd700 1 -- 172.21.15.154:0/1481276940 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a90023410 con 0x7f4a980724b0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.420+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9810d800 con 0x7f4a980724b0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.421+0000 7f4a86ffd700 1 -- 172.21.15.154:0/1481276940 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f4a9000f070 con 0x7f4a980724b0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.421+0000 7f4a86ffd700 1 --2- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4a7c02feb0 0x7f4a7c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.421+0000 7f4a86ffd700 1 -- 172.21.15.154:0/1481276940 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a90042180 con 0x7f4a980724b0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.421+0000 7f4a95d9b700 1 --2- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4a7c02feb0 0x7f4a7c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.421+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a98101dc0 con 0x7f4a980724b0 2024-01-16T17:10:58.405 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.422+0000 7f4a95d9b700 1 --2- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4a7c02feb0 0x7f4a7c032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4a8c0096a0 tx=0x7f4a8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:58.407 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.426+0000 7f4a86ffd700 1 -- 172.21.15.154:0/1481276940 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a90021320 con 0x7f4a980724b0 2024-01-16T17:10:58.572 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.590+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 --> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4a98101f80 con 0x7f4a7c02feb0 2024-01-16T17:10:58.787 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.805+0000 7f4a86ffd700 1 -- 172.21.15.154:0/1481276940 <== mgr.14162 v2:172.21.15.154:6800/2708438664 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f4a98101f80 con 0x7f4a7c02feb0 2024-01-16T17:10:58.787 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$9jVH6oa0q2unnDdAakLoPesT0BHjggl3w4XP3RkXsBXY.nPgIQZ46", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1705425058, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-16T17:10:58.789 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4a7c02feb0 msgr2=0x7f4a7c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 --2- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4a7c02feb0 0x7f4a7c032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4a8c0096a0 tx=0x7f4a8c006b40).stop 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 msgr2=0x7f4a9810cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 --2- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a9810cd00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f4a90000f80 tx=0x7f4a900092d0).stop 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 shutdown_connections 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 --2- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4a7c02feb0 0x7f4a7c032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 --2- 172.21.15.154:0/1481276940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a980724b0 0x7f4a9810cd00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 >> 172.21.15.154:0/1481276940 conn(0x7f4a980f74c0 msgr2=0x7f4a980f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 shutdown_connections 2024-01-16T17:10:58.790 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:58.807+0000 7f4a9cf47700 1 -- 172.21.15.154:0/1481276940 wait complete. 2024-01-16T17:10:59.076 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:58 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:10:57.786755+0000 mon.smithi154 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi154.sllivz(active, since 2s) 2024-01-16T17:10:59.076 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:58 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:58.592105+0000 mgr.smithi154.sllivz (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:10:59.149 INFO:teuthology.orchestra.run.smithi154.stderr:Fetching dashboard port number... 2024-01-16T17:10:59.149 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmplbjoqvqr:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpq82nhrg6:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.512+0000 7f243fd95700 1 Processor -- start 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.512+0000 7f243fd95700 1 -- start start 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243fd95700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f24380fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243fd95700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24380fb490 con 0x7f24380fab80 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243db31700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f24380fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243db31700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f24380fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47582/0 (socket says 172.21.15.154:47582) 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243db31700 1 -- 172.21.15.154:0/220559858 learned_addr learned my addr 172.21.15.154:0/220559858 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243db31700 1 -- 172.21.15.154:0/220559858 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24380fb5d0 con 0x7f24380fab80 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243db31700 1 --2- 172.21.15.154:0/220559858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f24380fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f242800b5f0 tx=0x7f2428006ee0).ready entity=mon.0 client_cookie=73a466a48e0d97ea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243cb2f700 1 -- 172.21.15.154:0/220559858 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f242800d600 con 0x7f24380fab80 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243cb2f700 1 -- 172.21.15.154:0/220559858 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f242800dbd0 con 0x7f24380fab80 2024-01-16T17:10:59.496 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.513+0000 7f243cb2f700 1 -- 172.21.15.154:0/220559858 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2428011c00 con 0x7f24380fab80 2024-01-16T17:10:59.497 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.514+0000 7f243fd95700 1 -- 172.21.15.154:0/220559858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 msgr2=0x7f24380fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:59.497 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.514+0000 7f243fd95700 1 --2- 172.21.15.154:0/220559858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f24380fafa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f242800b5f0 tx=0x7f2428006ee0).stop 2024-01-16T17:10:59.497 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.514+0000 7f243fd95700 1 -- 172.21.15.154:0/220559858 shutdown_connections 2024-01-16T17:10:59.499 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.514+0000 7f243fd95700 1 --2- 172.21.15.154:0/220559858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f24380fafa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:59.499 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.514+0000 7f243fd95700 1 -- 172.21.15.154:0/220559858 >> 172.21.15.154:0/220559858 conn(0x7f24380f63c0 msgr2=0x7f24380f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:59.499 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.514+0000 7f243fd95700 1 -- 172.21.15.154:0/220559858 shutdown_connections 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.514+0000 7f243fd95700 1 -- 172.21.15.154:0/220559858 wait complete. 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243fd95700 1 Processor -- start 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243fd95700 1 -- start start 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243fd95700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f2438100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243fd95700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24380ff290 con 0x7f24380fab80 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243db31700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f2438100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243db31700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f2438100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47584/0 (socket says 172.21.15.154:47584) 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243db31700 1 -- 172.21.15.154:0/2219958751 learned_addr learned my addr 172.21.15.154:0/2219958751 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243db31700 1 -- 172.21.15.154:0/2219958751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242800b200 con 0x7f24380fab80 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243db31700 1 --2- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f2438100e10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f242800b1d0 tx=0x7f2428006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f24267fc700 1 -- 172.21.15.154:0/2219958751 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f242801b410 con 0x7f24380fab80 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f24267fc700 1 -- 172.21.15.154:0/2219958751 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f242801b9e0 con 0x7f24380fab80 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f24267fc700 1 -- 172.21.15.154:0/2219958751 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f242801a840 con 0x7f24380fab80 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.515+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24380ff430 con 0x7f24380fab80 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.516+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24380ff840 con 0x7f24380fab80 2024-01-16T17:10:59.500 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.516+0000 7f24267fc700 1 -- 172.21.15.154:0/2219958751 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f242800f070 con 0x7f24380fab80 2024-01-16T17:10:59.501 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.516+0000 7f24267fc700 1 --2- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f242002ff00 0x7f24200323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:10:59.501 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.516+0000 7f24267fc700 1 -- 172.21.15.154:0/2219958751 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2428041490 con 0x7f24380fab80 2024-01-16T17:10:59.501 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.517+0000 7f243d330700 1 --2- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f242002ff00 0x7f24200323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:10:59.501 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.517+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2438059100 con 0x7f24380fab80 2024-01-16T17:10:59.501 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.517+0000 7f243d330700 1 --2- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f242002ff00 0x7f24200323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f24340096a0 tx=0x7f2434006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:10:59.503 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.521+0000 7f24267fc700 1 -- 172.21.15.154:0/2219958751 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2428016070 con 0x7f24380fab80 2024-01-16T17:10:59.652 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.670+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2438045ad0 con 0x7f24380fab80 2024-01-16T17:10:59.652 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.670+0000 7f24267fc700 1 -- 172.21.15.154:0/2219958751 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2428019950 con 0x7f24380fab80 2024-01-16T17:10:59.652 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 8443 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f242002ff00 msgr2=0x7f24200323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 --2- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f242002ff00 0x7f24200323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f24340096a0 tx=0x7f2434006b40).stop 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 msgr2=0x7f2438100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 --2- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f2438100e10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f242800b1d0 tx=0x7f2428006ee0).stop 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 shutdown_connections 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 --2- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f242002ff00 0x7f24200323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 --2- 172.21.15.154:0/2219958751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24380fab80 0x7f2438100e10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 >> 172.21.15.154:0/2219958751 conn(0x7f24380f63c0 msgr2=0x7f2438070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 shutdown_connections 2024-01-16T17:10:59.655 INFO:teuthology.orchestra.run.smithi154.stderr:/usr/bin/ceph: 2024-01-16T17:10:59.672+0000 7f243fd95700 1 -- 172.21.15.154:0/2219958751 wait complete. 2024-01-16T17:10:59.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:58.806358+0000 mon.smithi154 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:10:59.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:10:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:10:59.671642+0000 mon.smithi154 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.154:0/2219958751' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-16T17:11:00.108 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-16T17:11:00.116 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: disabled 2024-01-16T17:11:00.117 INFO:teuthology.orchestra.run.smithi154.stderr:Running command: systemctl is-active firewalld.service 2024-01-16T17:11:00.124 INFO:teuthology.orchestra.run.smithi154.stderr:systemctl: inactive 2024-01-16T17:11:00.124 INFO:teuthology.orchestra.run.smithi154.stderr:firewalld.service is not enabled 2024-01-16T17:11:00.124 INFO:teuthology.orchestra.run.smithi154.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr:Ceph Dashboard is now available at: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: URL: https://smithi154.front.sepia.ceph.com:8443/ 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: User: admin 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: Password: 8znwr5zxu0 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr:You can access the Ceph CLI with: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 059f8940-b492-11ee-95ae-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: ceph telemetry on 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr:For more information see: 2024-01-16T17:11:00.128 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:11:00.129 INFO:teuthology.orchestra.run.smithi154.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-16T17:11:00.129 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:11:00.129 INFO:teuthology.orchestra.run.smithi154.stderr:Bootstrap complete. 2024-01-16T17:11:00.129 INFO:teuthology.orchestra.run.smithi154.stderr:Releasing lock 140191401626312 on /run/cephadm/059f8940-b492-11ee-95ae-87774f69a715.lock 2024-01-16T17:11:00.129 INFO:teuthology.orchestra.run.smithi154.stderr:Lock 140191401626312 released on /run/cephadm/059f8940-b492-11ee-95ae-87774f69a715.lock 2024-01-16T17:11:00.160 INFO:tasks.cephadm:Fetching config... 2024-01-16T17:11:00.160 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:11:00.160 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-16T17:11:00.218 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-16T17:11:00.218 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:11:00.218 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-16T17:11:00.276 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-16T17:11:00.276 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:11:00.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/mon.smithi154/keyring of=/dev/stdout 2024-01-16T17:11:00.343 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-16T17:11:00.343 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-01-16T17:11:00.343 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-16T17:11:00.401 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-16T17:11:00.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVTdKBTk/l7yHsy1LAn3fkx1TBc5wMMM5me4zWd6oh8zvQaxKlzz+to1ciyb+hM5A52Tw7Ip0fkJRhhUGnuFHd90HtajdKJY0bz+W5+y1+7J4zPXnoR7p8hB77g0pPIfpojnrBl8c6ylc0Enx3Lu+m+pGB8U5KuFhRttwRttGm9OKK97TcW1F7CduC3UtoY1lGOoRimwVcx3GgJEeXcbeqinOEfxuazWeB1sDEDJqLIcymPyGsv3mAN3ulMxfRrXvO+h1pV/UPGiMneZmZSiiQ7WgTAjKYAzVbW1vcSxfAVZVSsPKGQYTobht5c/Fh5ijZAshwA6s+Zcx2n/UFgyPD3DeFYi+ahhApY/RS1GSKciLYTLn6+etkmS676cwlHPe1EsflDX1mu977o9uo9lkrr8ufSHQpSjtgr4sdARuuiPtc8+PvzkDLIwBqjwBCb0z+v7q9VYrAVuxJoieTCH8MPK0Yt1zOdHmNkt/sowTfM4XkLXahySuCYFgaloqe7uk= ceph-059f8940-b492-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-16T17:11:00.477 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVTdKBTk/l7yHsy1LAn3fkx1TBc5wMMM5me4zWd6oh8zvQaxKlzz+to1ciyb+hM5A52Tw7Ip0fkJRhhUGnuFHd90HtajdKJY0bz+W5+y1+7J4zPXnoR7p8hB77g0pPIfpojnrBl8c6ylc0Enx3Lu+m+pGB8U5KuFhRttwRttGm9OKK97TcW1F7CduC3UtoY1lGOoRimwVcx3GgJEeXcbeqinOEfxuazWeB1sDEDJqLIcymPyGsv3mAN3ulMxfRrXvO+h1pV/UPGiMneZmZSiiQ7WgTAjKYAzVbW1vcSxfAVZVSsPKGQYTobht5c/Fh5ijZAshwA6s+Zcx2n/UFgyPD3DeFYi+ahhApY/RS1GSKciLYTLn6+etkmS676cwlHPe1EsflDX1mu977o9uo9lkrr8ufSHQpSjtgr4sdARuuiPtc8+PvzkDLIwBqjwBCb0z+v7q9VYrAVuxJoieTCH8MPK0Yt1zOdHmNkt/sowTfM4XkLXahySuCYFgaloqe7uk= ceph-059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:11:00.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVTdKBTk/l7yHsy1LAn3fkx1TBc5wMMM5me4zWd6oh8zvQaxKlzz+to1ciyb+hM5A52Tw7Ip0fkJRhhUGnuFHd90HtajdKJY0bz+W5+y1+7J4zPXnoR7p8hB77g0pPIfpojnrBl8c6ylc0Enx3Lu+m+pGB8U5KuFhRttwRttGm9OKK97TcW1F7CduC3UtoY1lGOoRimwVcx3GgJEeXcbeqinOEfxuazWeB1sDEDJqLIcymPyGsv3mAN3ulMxfRrXvO+h1pV/UPGiMneZmZSiiQ7WgTAjKYAzVbW1vcSxfAVZVSsPKGQYTobht5c/Fh5ijZAshwA6s+Zcx2n/UFgyPD3DeFYi+ahhApY/RS1GSKciLYTLn6+etkmS676cwlHPe1EsflDX1mu977o9uo9lkrr8ufSHQpSjtgr4sdARuuiPtc8+PvzkDLIwBqjwBCb0z+v7q9VYrAVuxJoieTCH8MPK0Yt1zOdHmNkt/sowTfM4XkLXahySuCYFgaloqe7uk= ceph-059f8940-b492-11ee-95ae-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-16T17:11:00.531 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVTdKBTk/l7yHsy1LAn3fkx1TBc5wMMM5me4zWd6oh8zvQaxKlzz+to1ciyb+hM5A52Tw7Ip0fkJRhhUGnuFHd90HtajdKJY0bz+W5+y1+7J4zPXnoR7p8hB77g0pPIfpojnrBl8c6ylc0Enx3Lu+m+pGB8U5KuFhRttwRttGm9OKK97TcW1F7CduC3UtoY1lGOoRimwVcx3GgJEeXcbeqinOEfxuazWeB1sDEDJqLIcymPyGsv3mAN3ulMxfRrXvO+h1pV/UPGiMneZmZSiiQ7WgTAjKYAzVbW1vcSxfAVZVSsPKGQYTobht5c/Fh5ijZAshwA6s+Zcx2n/UFgyPD3DeFYi+ahhApY/RS1GSKciLYTLn6+etkmS676cwlHPe1EsflDX1mu977o9uo9lkrr8ufSHQpSjtgr4sdARuuiPtc8+PvzkDLIwBqjwBCb0z+v7q9VYrAVuxJoieTCH8MPK0Yt1zOdHmNkt/sowTfM4XkLXahySuCYFgaloqe7uk= ceph-059f8940-b492-11ee-95ae-87774f69a715 2024-01-16T17:11:00.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 -- 172.21.15.154:0/3268292310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 msgr2=0x7f40040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 --2- 172.21.15.154:0/3268292310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 0x7f40040f5830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f3ff4009fb0 tx=0x7f3ff4009b30).stop 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 -- 172.21.15.154:0/3268292310 shutdown_connections 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 --2- 172.21.15.154:0/3268292310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 0x7f40040f5830 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 -- 172.21.15.154:0/3268292310 >> 172.21.15.154:0/3268292310 conn(0x7f40040f0b70 msgr2=0x7f40040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 -- 172.21.15.154:0/3268292310 shutdown_connections 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 -- 172.21.15.154:0/3268292310 wait complete. 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 Processor -- start 2024-01-16T17:11:01.667 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 -- start start 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 0x7f4004109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.684+0000 7f400c159700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4004060220 con 0x7f40040f5410 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f4009ef5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 0x7f4004109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f4009ef5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 0x7f4004109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47592/0 (socket says 172.21.15.154:47592) 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f4009ef5700 1 -- 172.21.15.154:0/1365503596 learned_addr learned my addr 172.21.15.154:0/1365503596 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f4009ef5700 1 -- 172.21.15.154:0/1365503596 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff4005040 con 0x7f40040f5410 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f4009ef5700 1 --2- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 0x7f4004109410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3ff4004dd0 tx=0x7f3ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f3ffa7fc700 1 -- 172.21.15.154:0/1365503596 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff4009320 con 0x7f40040f5410 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f3ffa7fc700 1 -- 172.21.15.154:0/1365503596 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3ff4013680 con 0x7f40040f5410 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f3ffa7fc700 1 -- 172.21.15.154:0/1365503596 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ff401ecb0 con 0x7f40040f5410 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4004060420 con 0x7f40040f5410 2024-01-16T17:11:01.668 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.685+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4004109ac0 con 0x7f40040f5410 2024-01-16T17:11:01.669 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.686+0000 7f3ffa7fc700 1 -- 172.21.15.154:0/1365503596 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3ff401e420 con 0x7f40040f5410 2024-01-16T17:11:01.669 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.686+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4004059100 con 0x7f40040f5410 2024-01-16T17:11:01.669 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.686+0000 7f3ffa7fc700 1 --2- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3ff002ff00 0x7f3ff00323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:01.669 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.686+0000 7f3ffa7fc700 1 -- 172.21.15.154:0/1365503596 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ff4040c30 con 0x7f40040f5410 2024-01-16T17:11:01.669 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.686+0000 7f40096f4700 1 --2- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3ff002ff00 0x7f3ff00323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:01.669 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.686+0000 7f40096f4700 1 --2- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3ff002ff00 0x7f3ff00323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f40000096a0 tx=0x7f4000006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:01.672 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.690+0000 7f3ffa7fc700 1 -- 172.21.15.154:0/1365503596 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ff4047080 con 0x7f40040f5410 2024-01-16T17:11:01.797 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.815+0000 7f3ffa7fc700 1 -- 172.21.15.154:0/1365503596 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ff4014940 con 0x7f40040f5410 2024-01-16T17:11:01.823 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.842+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f40040fae50 con 0x7f40040f5410 2024-01-16T17:11:01.826 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.844+0000 7f3ffa7fc700 1 -- 172.21.15.154:0/1365503596 <== mon.0 v2:172.21.15.154:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v10)=0 v10) v1 ==== 125+0+0 (secure 0 0 0) 0x7f3ff401dba0 con 0x7f40040f5410 2024-01-16T17:11:01.828 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.846+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3ff002ff00 msgr2=0x7f3ff00323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:01.828 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.846+0000 7f400c159700 1 --2- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3ff002ff00 0x7f3ff00323c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f40000096a0 tx=0x7f4000006b40).stop 2024-01-16T17:11:01.829 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.846+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 msgr2=0x7f4004109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:01.829 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.846+0000 7f400c159700 1 --2- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 0x7f4004109410 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f3ff4004dd0 tx=0x7f3ff400b040).stop 2024-01-16T17:11:01.829 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.846+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 shutdown_connections 2024-01-16T17:11:01.829 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.846+0000 7f400c159700 1 --2- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3ff002ff00 0x7f3ff00323c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:01.829 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.846+0000 7f400c159700 1 --2- 172.21.15.154:0/1365503596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40040f5410 0x7f4004109410 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:01.829 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.846+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 >> 172.21.15.154:0/1365503596 conn(0x7f40040f0b70 msgr2=0x7f40040f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:01.829 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.847+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 shutdown_connections 2024-01-16T17:11:01.829 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:01.847+0000 7f400c159700 1 -- 172.21.15.154:0/1365503596 wait complete. 2024-01-16T17:11:02.211 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-01-16T17:11:02.211 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-01-16T17:11:02.211 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-01-16T17:11:02.233 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-01-16T17:11:02.233 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-16T17:11:02.294 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-01-16T17:11:02.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph orch host add smithi190 2024-01-16T17:11:03.153 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:02 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:01.818208+0000 mon.smithi154 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi154.sllivz(active, since 6s) 2024-01-16T17:11:03.153 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:02 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:01.846010+0000 mon.smithi154 (mon.0) 108 : audit [INF] from='client.? 172.21.15.154:0/1365503596' entity='client.admin' 2024-01-16T17:11:05.103 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:04 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:03.782959+0000 mon.smithi154 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:11:05.103 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:04 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:03.999534+0000 mon.smithi154 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:11:05.103 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:04 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:04.597280+0000 mon.smithi154 (mon.0) 111 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:11:08.210 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:06.937578+0000 mon.smithi154 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:11:08.210 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:07.210198+0000 mon.smithi154 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:11:08.210 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:07.212393+0000 mon.smithi154 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:11:08.210 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:11:07.218690+0000 mgr.smithi154.sllivz (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi154 on smithi154 2024-01-16T17:11:17.210 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:16 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:15.785362+0000 mgr.smithi154.sllivz (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:19.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:17.785708+0000 mgr.smithi154.sllivz (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:21.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:19.786021+0000 mgr.smithi154.sllivz (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:23.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:21.786356+0000 mgr.smithi154.sllivz (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:25.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.271+0000 7fa3c0342700 1 -- 172.21.15.190:0/660761858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 msgr2=0x7fa3b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:25.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.271+0000 7fa3c0342700 1 --2- 172.21.15.190:0/660761858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 0x7fa3b80f6930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa3a8009fb0 tx=0x7fa3a8009b30).stop 2024-01-16T17:11:25.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.271+0000 7fa3c0342700 1 -- 172.21.15.190:0/660761858 shutdown_connections 2024-01-16T17:11:25.254 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.271+0000 7fa3c0342700 1 --2- 172.21.15.190:0/660761858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 0x7fa3b80f6930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.271+0000 7fa3c0342700 1 -- 172.21.15.190:0/660761858 >> 172.21.15.190:0/660761858 conn(0x7fa3b80f1c70 msgr2=0x7fa3b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.271+0000 7fa3c0342700 1 -- 172.21.15.190:0/660761858 shutdown_connections 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.271+0000 7fa3c0342700 1 -- 172.21.15.190:0/660761858 wait complete. 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.272+0000 7fa3c0342700 1 Processor -- start 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.272+0000 7fa3c0342700 1 -- start start 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.272+0000 7fa3c0342700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 0x7fa3b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.272+0000 7fa3c0342700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8106c20 con 0x7fa3b80f6510 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.272+0000 7fa3be0de700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 0x7fa3b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.272+0000 7fa3be0de700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 0x7fa3b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46610/0 (socket says 172.21.15.190:46610) 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.272+0000 7fa3be0de700 1 -- 172.21.15.190:0/1782952991 learned_addr learned my addr 172.21.15.190:0/1782952991 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.273+0000 7fa3be0de700 1 -- 172.21.15.190:0/1782952991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a8005040 con 0x7fa3b80f6510 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.273+0000 7fa3be0de700 1 --2- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 0x7fa3b8106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa3a8013510 tx=0x7fa3a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.273+0000 7fa3aeffd700 1 -- 172.21.15.190:0/1782952991 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a8009320 con 0x7fa3b80f6510 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.273+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b8106e20 con 0x7fa3b80f6510 2024-01-16T17:11:25.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.273+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b81072b0 con 0x7fa3b80f6510 2024-01-16T17:11:25.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.273+0000 7fa3aeffd700 1 -- 172.21.15.190:0/1782952991 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3a8013f00 con 0x7fa3b80f6510 2024-01-16T17:11:25.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.273+0000 7fa3aeffd700 1 -- 172.21.15.190:0/1782952991 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a8014d60 con 0x7fa3b80f6510 2024-01-16T17:11:25.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.274+0000 7fa3aeffd700 1 -- 172.21.15.190:0/1782952991 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3a80095f0 con 0x7fa3b80f6510 2024-01-16T17:11:25.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.274+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b8045ad0 con 0x7fa3b80f6510 2024-01-16T17:11:25.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.274+0000 7fa3aeffd700 1 --2- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3a402ff60 0x7fa3a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:25.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.274+0000 7fa3aeffd700 1 -- 172.21.15.190:0/1782952991 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3a8040ff0 con 0x7fa3b80f6510 2024-01-16T17:11:25.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.274+0000 7fa3bd8dd700 1 --2- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3a402ff60 0x7fa3a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:25.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.274+0000 7fa3bd8dd700 1 --2- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3a402ff60 0x7fa3a4032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa3b40096a0 tx=0x7fa3b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:25.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.279+0000 7fa3aeffd700 1 -- 172.21.15.190:0/1782952991 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3a8014420 con 0x7fa3b80f6510 2024-01-16T17:11:25.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:25.449+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 --> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3b80f2e70 con 0x7fa3a402ff60 2024-01-16T17:11:25.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:23.786636+0000 mgr.smithi154.sllivz (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:26.034 INFO:teuthology.orchestra.run.smithi190.stdout:Added host 'smithi190' 2024-01-16T17:11:26.035 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.052+0000 7fa3aeffd700 1 -- 172.21.15.190:0/1782952991 <== mgr.14162 v2:172.21.15.154:6800/2708438664 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fa3b80f2e70 con 0x7fa3a402ff60 2024-01-16T17:11:26.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3a402ff60 msgr2=0x7fa3a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:26.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 --2- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3a402ff60 0x7fa3a4032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa3b40096a0 tx=0x7fa3b4006b40).stop 2024-01-16T17:11:26.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 msgr2=0x7fa3b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:26.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 --2- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 0x7fa3b8106730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa3a8013510 tx=0x7fa3a800b040).stop 2024-01-16T17:11:26.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 shutdown_connections 2024-01-16T17:11:26.036 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 --2- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3a402ff60 0x7fa3a4032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:26.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 --2- 172.21.15.190:0/1782952991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3b80f6510 0x7fa3b8106730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:26.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 >> 172.21.15.190:0/1782952991 conn(0x7fa3b80f1c70 msgr2=0x7fa3b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:26.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 shutdown_connections 2024-01-16T17:11:26.037 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:26.055+0000 7fa3c0342700 1 -- 172.21.15.190:0/1782952991 wait complete. 2024-01-16T17:11:26.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph orch host ls --format=json 2024-01-16T17:11:26.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:25.451526+0000 mgr.smithi154.sllivz (mgr.14162) 11 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:11:26.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:26.052888+0000 mon.smithi154 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:11:27.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.456+0000 7f83fdf6a700 1 -- 172.21.15.190:0/994430995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 msgr2=0x7f83f80f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.456+0000 7f83fdf6a700 1 --2- 172.21.15.190:0/994430995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 0x7f83f80f4bd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f83e0009fb0 tx=0x7f83e0009b30).stop 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.456+0000 7f83fdf6a700 1 -- 172.21.15.190:0/994430995 shutdown_connections 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.456+0000 7f83fdf6a700 1 --2- 172.21.15.190:0/994430995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 0x7f83f80f4bd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.456+0000 7f83fdf6a700 1 -- 172.21.15.190:0/994430995 >> 172.21.15.190:0/994430995 conn(0x7f83f80f02b0 msgr2=0x7f83f80f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.456+0000 7f83fdf6a700 1 -- 172.21.15.190:0/994430995 shutdown_connections 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.456+0000 7f83fdf6a700 1 -- 172.21.15.190:0/994430995 wait complete. 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.457+0000 7f83fdf6a700 1 Processor -- start 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.457+0000 7f83fdf6a700 1 -- start start 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.457+0000 7f83fdf6a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 0x7f83f81066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:27.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.457+0000 7f83fdf6a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f8106b90 con 0x7f83f80f47b0 2024-01-16T17:11:27.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83f77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 0x7f83f81066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:27.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83f77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 0x7f83f81066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46622/0 (socket says 172.21.15.190:46622) 2024-01-16T17:11:27.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83f77fe700 1 -- 172.21.15.190:0/4008260694 learned_addr learned my addr 172.21.15.190:0/4008260694 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:27.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83f77fe700 1 -- 172.21.15.190:0/4008260694 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0005040 con 0x7f83f80f47b0 2024-01-16T17:11:27.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83f77fe700 1 --2- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 0x7f83f81066a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f83e0013510 tx=0x7f83e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:27.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83effff700 1 -- 172.21.15.190:0/4008260694 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83e0009320 con 0x7f83f80f47b0 2024-01-16T17:11:27.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f8106d90 con 0x7f83f80f47b0 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83effff700 1 -- 172.21.15.190:0/4008260694 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83e0013f00 con 0x7f83f80f47b0 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83effff700 1 -- 172.21.15.190:0/4008260694 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83e0014d90 con 0x7f83f80f47b0 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.458+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f8107220 con 0x7f83f80f47b0 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.459+0000 7f83effff700 1 -- 172.21.15.190:0/4008260694 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83e00263a0 con 0x7f83f80f47b0 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.459+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f8045ad0 con 0x7f83f80f47b0 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.459+0000 7f83effff700 1 --2- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f83e402ff60 0x7f83e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.459+0000 7f83effff700 1 -- 172.21.15.190:0/4008260694 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83e00413b0 con 0x7f83f80f47b0 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.459+0000 7f83f6ffd700 1 --2- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f83e402ff60 0x7f83e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:27.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.460+0000 7f83f6ffd700 1 --2- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f83e402ff60 0x7f83e4032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f83e80096a0 tx=0x7f83e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:27.445 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.464+0000 7f83effff700 1 -- 172.21.15.190:0/4008260694 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83e00126f0 con 0x7f83f80f47b0 2024-01-16T17:11:27.613 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.632+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 --> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83f80f1490 con 0x7f83e402ff60 2024-01-16T17:11:27.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.634+0000 7f83effff700 1 -- 172.21.15.190:0/4008260694 <== mgr.14162 v2:172.21.15.154:6800/2708438664 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f83f80f1490 con 0x7f83e402ff60 2024-01-16T17:11:27.615 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:27.615 INFO:teuthology.orchestra.run.smithi190.stdout:[{"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}, {"addr": "smithi190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f83e402ff60 msgr2=0x7f83e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 --2- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f83e402ff60 0x7f83e4032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f83e80096a0 tx=0x7f83e8006b40).stop 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 msgr2=0x7f83f81066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 --2- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 0x7f83f81066a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f83e0013510 tx=0x7f83e000b040).stop 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 shutdown_connections 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 --2- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f83e402ff60 0x7f83e4032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 --2- 172.21.15.190:0/4008260694 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83f80f47b0 0x7f83f81066a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 >> 172.21.15.190:0/4008260694 conn(0x7f83f80f02b0 msgr2=0x7f83f80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 shutdown_connections 2024-01-16T17:11:27.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:27.635+0000 7f83fdf6a700 1 -- 172.21.15.190:0/4008260694 wait complete. 2024-01-16T17:11:27.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:25.787045+0000 mgr.smithi154.sllivz (mgr.14162) 12 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:27.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:11:26.053324+0000 mgr.smithi154.sllivz (mgr.14162) 13 : cephadm [INF] Added host smithi190 2024-01-16T17:11:27.986 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-16T17:11:27.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph osd crush tunables default 2024-01-16T17:11:28.643 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:28 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:27.634299+0000 mgr.smithi154.sllivz (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-16T17:11:29.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.152+0000 7f32a51c8700 1 -- 172.21.15.154:0/2375864278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 msgr2=0x7f32a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:29.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.152+0000 7f32a51c8700 1 --2- 172.21.15.154:0/2375864278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 0x7f32a00f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3290009fb0 tx=0x7f3290009b30).stop 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.152+0000 7f32a51c8700 1 -- 172.21.15.154:0/2375864278 shutdown_connections 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.152+0000 7f32a51c8700 1 --2- 172.21.15.154:0/2375864278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 0x7f32a00f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.152+0000 7f32a51c8700 1 -- 172.21.15.154:0/2375864278 >> 172.21.15.154:0/2375864278 conn(0x7f32a00f0b70 msgr2=0x7f32a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.152+0000 7f32a51c8700 1 -- 172.21.15.154:0/2375864278 shutdown_connections 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.152+0000 7f32a51c8700 1 -- 172.21.15.154:0/2375864278 wait complete. 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.153+0000 7f32a51c8700 1 Processor -- start 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.153+0000 7f32a51c8700 1 -- start start 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.153+0000 7f32a51c8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 0x7f32a0106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.153+0000 7f32a51c8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a0106c00 con 0x7f32a00f5410 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.153+0000 7f329ed9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 0x7f32a0106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.153+0000 7f329ed9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 0x7f32a0106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:47524/0 (socket says 172.21.15.154:47524) 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.153+0000 7f329ed9d700 1 -- 172.21.15.154:0/3680372023 learned_addr learned my addr 172.21.15.154:0/3680372023 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-01-16T17:11:29.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.154+0000 7f329ed9d700 1 -- 172.21.15.154:0/3680372023 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3290005040 con 0x7f32a00f5410 2024-01-16T17:11:29.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.154+0000 7f329ed9d700 1 --2- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 0x7f32a0106710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3290013510 tx=0x7f329000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:29.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.154+0000 7f32877fe700 1 -- 172.21.15.154:0/3680372023 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3290009320 con 0x7f32a00f5410 2024-01-16T17:11:29.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.154+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a0106e00 con 0x7f32a00f5410 2024-01-16T17:11:29.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.154+0000 7f32877fe700 1 -- 172.21.15.154:0/3680372023 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3290013f00 con 0x7f32a00f5410 2024-01-16T17:11:29.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.154+0000 7f32877fe700 1 -- 172.21.15.154:0/3680372023 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3290014d90 con 0x7f32a00f5410 2024-01-16T17:11:29.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.154+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a0107290 con 0x7f32a00f5410 2024-01-16T17:11:29.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.155+0000 7f32877fe700 1 -- 172.21.15.154:0/3680372023 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32900263a0 con 0x7f32a00f5410 2024-01-16T17:11:29.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.155+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a0045ad0 con 0x7f32a00f5410 2024-01-16T17:11:29.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.155+0000 7f32877fe700 1 --2- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f328802ff60 0x7f3288032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:29.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.155+0000 7f32877fe700 1 -- 172.21.15.154:0/3680372023 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3290041300 con 0x7f32a00f5410 2024-01-16T17:11:29.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.155+0000 7f329e59c700 1 --2- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f328802ff60 0x7f3288032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:29.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.155+0000 7f329e59c700 1 --2- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f328802ff60 0x7f3288032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f32940096a0 tx=0x7f3294006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:29.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.159+0000 7f32877fe700 1 -- 172.21.15.154:0/3680372023 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32900126f0 con 0x7f32a00f5410 2024-01-16T17:11:29.295 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.314+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f32a0108140 con 0x7f32a00f5410 2024-01-16T17:11:29.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.394+0000 7f32877fe700 1 -- 172.21.15.154:0/3680372023 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3290012330 con 0x7f32a00f5410 2024-01-16T17:11:29.377 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f328802ff60 msgr2=0x7f3288032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 --2- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f328802ff60 0x7f3288032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f32940096a0 tx=0x7f3294006b40).stop 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 msgr2=0x7f32a0106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 --2- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 0x7f32a0106710 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3290013510 tx=0x7f329000b040).stop 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 shutdown_connections 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 --2- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f328802ff60 0x7f3288032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 --2- 172.21.15.154:0/3680372023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32a00f5410 0x7f32a0106710 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 >> 172.21.15.154:0/3680372023 conn(0x7f32a00f0b70 msgr2=0x7f32a005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 shutdown_connections 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:2024-01-16T17:11:29.396+0000 7f32a51c8700 1 -- 172.21.15.154:0/3680372023 wait complete. 2024-01-16T17:11:29.378 INFO:teuthology.orchestra.run.smithi154.stderr:adjusted tunables profile to default 2024-01-16T17:11:29.654 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:27.787432+0000 mgr.smithi154.sllivz (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:29.654 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:29.315701+0000 mon.smithi154 (mon.0) 116 : audit [INF] from='client.? 172.21.15.154:0/3680372023' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-16T17:11:29.746 INFO:tasks.cephadm:Adding mon.smithi154 on smithi154 2024-01-16T17:11:29.746 INFO:tasks.cephadm:Adding mon.smithi190 on smithi190 2024-01-16T17:11:29.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph orch apply mon '2;smithi154:172.21.15.154=smithi154;smithi190:172.21.15.190=smithi190' 2024-01-16T17:11:30.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:30 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:29.395764+0000 mon.smithi154 (mon.0) 117 : audit [INF] from='client.? 172.21.15.154:0/3680372023' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-16T17:11:30.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:30 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:29.395805+0000 mon.smithi154 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-16T17:11:30.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.924+0000 7f6bcff35700 1 -- 172.21.15.190:0/1788512864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 msgr2=0x7f6bc80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:30.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.924+0000 7f6bcff35700 1 --2- 172.21.15.190:0/1788512864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 0x7f6bc80f90f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f6bb8009fb0 tx=0x7f6bb8009b30).stop 2024-01-16T17:11:30.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.924+0000 7f6bcff35700 1 -- 172.21.15.190:0/1788512864 shutdown_connections 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.924+0000 7f6bcff35700 1 --2- 172.21.15.190:0/1788512864 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 0x7f6bc80f90f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.924+0000 7f6bcff35700 1 -- 172.21.15.190:0/1788512864 >> 172.21.15.190:0/1788512864 conn(0x7f6bc80f0b70 msgr2=0x7f6bc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.924+0000 7f6bcff35700 1 -- 172.21.15.190:0/1788512864 shutdown_connections 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.924+0000 7f6bcff35700 1 -- 172.21.15.190:0/1788512864 wait complete. 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.925+0000 7f6bcff35700 1 Processor -- start 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.925+0000 7f6bcff35700 1 -- start start 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.925+0000 7f6bcff35700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 0x7f6bc81066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.925+0000 7f6bcff35700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc8106ba0 con 0x7f6bc8060280 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.925+0000 7f6bcdcd1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 0x7f6bc81066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.925+0000 7f6bcdcd1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 0x7f6bc81066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46646/0 (socket says 172.21.15.190:46646) 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.925+0000 7f6bcdcd1700 1 -- 172.21.15.190:0/3261172558 learned_addr learned my addr 172.21.15.190:0/3261172558 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.925+0000 7f6bcdcd1700 1 -- 172.21.15.190:0/3261172558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb8005040 con 0x7f6bc8060280 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.926+0000 7f6bcdcd1700 1 --2- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 0x7f6bc81066b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6bb8013510 tx=0x7f6bb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.926+0000 7f6bbe7fc700 1 -- 172.21.15.190:0/3261172558 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb8009320 con 0x7f6bc8060280 2024-01-16T17:11:30.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.926+0000 7f6bbe7fc700 1 -- 172.21.15.190:0/3261172558 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bb8013f00 con 0x7f6bc8060280 2024-01-16T17:11:30.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.926+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc8106da0 con 0x7f6bc8060280 2024-01-16T17:11:30.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.926+0000 7f6bbe7fc700 1 -- 172.21.15.190:0/3261172558 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb8014dc0 con 0x7f6bc8060280 2024-01-16T17:11:30.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.926+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc8107230 con 0x7f6bc8060280 2024-01-16T17:11:30.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.927+0000 7f6bbe7fc700 1 -- 172.21.15.190:0/3261172558 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bb80095f0 con 0x7f6bc8060280 2024-01-16T17:11:30.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.927+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc8045ad0 con 0x7f6bc8060280 2024-01-16T17:11:30.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.927+0000 7f6bbe7fc700 1 --2- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bb402ff60 0x7f6bb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:30.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.927+0000 7f6bbe7fc700 1 -- 172.21.15.190:0/3261172558 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bb80410c0 con 0x7f6bc8060280 2024-01-16T17:11:30.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.927+0000 7f6bcd4d0700 1 --2- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bb402ff60 0x7f6bb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:30.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.927+0000 7f6bcd4d0700 1 --2- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bb402ff60 0x7f6bb4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6bc40096a0 tx=0x7f6bc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:30.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:30.931+0000 7f6bbe7fc700 1 -- 172.21.15.190:0/3261172558 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bb8014420 con 0x7f6bc8060280 2024-01-16T17:11:31.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.109+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 --> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi154:172.21.15.154=smithi154;smithi190:172.21.15.190=smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bc80f1d20 con 0x7f6bb402ff60 2024-01-16T17:11:31.095 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.115+0000 7f6bbe7fc700 1 -- 172.21.15.190:0/3261172558 <== mgr.14162 v2:172.21.15.154:6800/2708438664 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6bc80f1d20 con 0x7f6bb402ff60 2024-01-16T17:11:31.096 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2024-01-16T17:11:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bb402ff60 msgr2=0x7f6bb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 --2- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bb402ff60 0x7f6bb4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6bc40096a0 tx=0x7f6bc4006b40).stop 2024-01-16T17:11:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 msgr2=0x7f6bc81066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 --2- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 0x7f6bc81066b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6bb8013510 tx=0x7f6bb800b040).stop 2024-01-16T17:11:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 shutdown_connections 2024-01-16T17:11:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 --2- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bb402ff60 0x7f6bb4032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 --2- 172.21.15.190:0/3261172558 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bc8060280 0x7f6bc81066b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 >> 172.21.15.190:0/3261172558 conn(0x7f6bc80f0b70 msgr2=0x7f6bc80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:31.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 shutdown_connections 2024-01-16T17:11:31.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:31.116+0000 7f6bcff35700 1 -- 172.21.15.190:0/3261172558 wait complete. 2024-01-16T17:11:31.460 DEBUG:teuthology.orchestra.run.smithi190:mon.smithi190> sudo journalctl -f -n 0 -u ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi190.service 2024-01-16T17:11:31.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:31.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:31.491 INFO:journalctl@ceph.mon.smithi190.smithi190.stdout:-- Logs begin at Tue 2024-01-16 16:57:03 UTC. -- 2024-01-16T17:11:31.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:29.787760+0000 mgr.smithi154.sllivz (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:31.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:31.114858+0000 mon.smithi154 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:11:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.624+0000 7fe26f0d1700 1 -- 172.21.15.190:0/2762182221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680f5410 msgr2=0x7fe2680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.624+0000 7fe26f0d1700 1 --2- 172.21.15.190:0/2762182221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680f5410 0x7fe2680f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe25c009fb0 tx=0x7fe25c009b30).stop 2024-01-16T17:11:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.624+0000 7fe26f0d1700 1 -- 172.21.15.190:0/2762182221 shutdown_connections 2024-01-16T17:11:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.624+0000 7fe26f0d1700 1 --2- 172.21.15.190:0/2762182221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680f5410 0x7fe2680f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.624+0000 7fe26f0d1700 1 -- 172.21.15.190:0/2762182221 >> 172.21.15.190:0/2762182221 conn(0x7fe2680f0b70 msgr2=0x7fe2680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:32.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.624+0000 7fe26f0d1700 1 -- 172.21.15.190:0/2762182221 shutdown_connections 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.624+0000 7fe26f0d1700 1 -- 172.21.15.190:0/2762182221 wait complete. 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.624+0000 7fe26f0d1700 1 Processor -- start 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.625+0000 7fe26f0d1700 1 -- start start 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.625+0000 7fe26f0d1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680601e0 0x7fe2681095a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.625+0000 7fe26f0d1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe268109a90 con 0x7fe2680601e0 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.625+0000 7fe26ce6d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680601e0 0x7fe2681095a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.625+0000 7fe26ce6d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680601e0 0x7fe2681095a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:42112/0 (socket says 172.21.15.190:42112) 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.625+0000 7fe26ce6d700 1 -- 172.21.15.190:0/3181761096 learned_addr learned my addr 172.21.15.190:0/3181761096 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.625+0000 7fe26ce6d700 1 -- 172.21.15.190:0/3181761096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c005040 con 0x7fe2680601e0 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.625+0000 7fe26ce6d700 1 --2- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680601e0 0x7fe2681095a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe25c009f80 tx=0x7fe25c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.626+0000 7fe2657fa700 1 -- 172.21.15.190:0/3181761096 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25c0136c0 con 0x7fe2680601e0 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.626+0000 7fe2657fa700 1 -- 172.21.15.190:0/3181761096 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe25c013e70 con 0x7fe2680601e0 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.626+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe268109bd0 con 0x7fe2680601e0 2024-01-16T17:11:32.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.626+0000 7fe2657fa700 1 -- 172.21.15.190:0/3181761096 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25c014420 con 0x7fe2680601e0 2024-01-16T17:11:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.626+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26810a000 con 0x7fe2680601e0 2024-01-16T17:11:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.626+0000 7fe2657fa700 1 -- 172.21.15.190:0/3181761096 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe25c025060 con 0x7fe2680601e0 2024-01-16T17:11:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.627+0000 7fe2657fa700 1 --2- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe25802ff60 0x7fe258032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:32.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.627+0000 7fe2657fa700 1 -- 172.21.15.190:0/3181761096 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe25c040ec0 con 0x7fe2680601e0 2024-01-16T17:11:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.627+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe268045ad0 con 0x7fe2680601e0 2024-01-16T17:11:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.627+0000 7fe267fff700 1 --2- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe25802ff60 0x7fe258032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:32.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.627+0000 7fe267fff700 1 --2- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe25802ff60 0x7fe258032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe2540096a0 tx=0x7fe254006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:32.613 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.631+0000 7fe2657fa700 1 -- 172.21.15.190:0/3181761096 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe25c014580 con 0x7fe2680601e0 2024-01-16T17:11:32.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:32 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:31.110857+0000 mgr.smithi154.sllivz (mgr.14162) 17 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi154:172.21.15.154=smithi154;smithi190:172.21.15.190=smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-01-16T17:11:32.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:32 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:11:31.112151+0000 mgr.smithi154.sllivz (mgr.14162) 18 : cephadm [INF] Saving service mon spec with placement smithi154:172.21.15.154=smithi154;smithi190:172.21.15.190=smithi190;count:2 2024-01-16T17:11:32.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.833+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2680fae60 con 0x7fe2680601e0 2024-01-16T17:11:32.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.834+0000 7fe2657fa700 1 -- 172.21.15.190:0/3181761096 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe25c01cb30 con 0x7fe2680601e0 2024-01-16T17:11:32.816 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:32.816 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:11:32.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.835+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe25802ff60 msgr2=0x7fe258032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:32.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.835+0000 7fe26f0d1700 1 --2- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe25802ff60 0x7fe258032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe2540096a0 tx=0x7fe254006b40).stop 2024-01-16T17:11:32.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.835+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680601e0 msgr2=0x7fe2681095a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:32.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.835+0000 7fe26f0d1700 1 --2- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680601e0 0x7fe2681095a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe25c009f80 tx=0x7fe25c009b30).stop 2024-01-16T17:11:32.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.835+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 shutdown_connections 2024-01-16T17:11:32.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.835+0000 7fe26f0d1700 1 --2- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe25802ff60 0x7fe258032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:32.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.835+0000 7fe26f0d1700 1 --2- 172.21.15.190:0/3181761096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2680601e0 0x7fe2681095a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:32.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.835+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 >> 172.21.15.190:0/3181761096 conn(0x7fe2680f0b70 msgr2=0x7fe2680f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:32.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.836+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 shutdown_connections 2024-01-16T17:11:32.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:32.836+0000 7fe26f0d1700 1 -- 172.21.15.190:0/3181761096 wait complete. 2024-01-16T17:11:32.819 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:11:33.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:31.788059+0000 mgr.smithi154.sllivz (mgr.14162) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:33.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:32.835790+0000 mon.smithi154 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.190:0/3181761096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:11:34.160 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:34.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:35.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.311+0000 7f5dfcae2700 1 -- 172.21.15.190:0/369796782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 msgr2=0x7f5df80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.311+0000 7f5dfcae2700 1 --2- 172.21.15.190:0/369796782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 0x7f5df80f8f20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f5de8009fb0 tx=0x7f5de8009b30).stop 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.312+0000 7f5dfcae2700 1 -- 172.21.15.190:0/369796782 shutdown_connections 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.312+0000 7f5dfcae2700 1 --2- 172.21.15.190:0/369796782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 0x7f5df80f8f20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.312+0000 7f5dfcae2700 1 -- 172.21.15.190:0/369796782 >> 172.21.15.190:0/369796782 conn(0x7f5df80f09a0 msgr2=0x7f5df80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.312+0000 7f5dfcae2700 1 -- 172.21.15.190:0/369796782 shutdown_connections 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.312+0000 7f5dfcae2700 1 -- 172.21.15.190:0/369796782 wait complete. 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.312+0000 7f5dfcae2700 1 Processor -- start 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.312+0000 7f5dfcae2700 1 -- start start 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.313+0000 7f5dfcae2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 0x7f5df8106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:35.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.313+0000 7f5dfcae2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df81069f0 con 0x7f5df80601e0 2024-01-16T17:11:35.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.313+0000 7f5df77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 0x7f5df8106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:35.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.313+0000 7f5df77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 0x7f5df8106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:42136/0 (socket says 172.21.15.190:42136) 2024-01-16T17:11:35.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.313+0000 7f5df77fe700 1 -- 172.21.15.190:0/3981845167 learned_addr learned my addr 172.21.15.190:0/3981845167 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:35.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.313+0000 7f5df77fe700 1 -- 172.21.15.190:0/3981845167 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de8005040 con 0x7f5df80601e0 2024-01-16T17:11:35.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.313+0000 7f5df77fe700 1 --2- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 0x7f5df8106500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5de8013510 tx=0x7f5de800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:35.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.313+0000 7f5ddffff700 1 -- 172.21.15.190:0/3981845167 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5de8009320 con 0x7f5df80601e0 2024-01-16T17:11:35.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.314+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df8106bf0 con 0x7f5df80601e0 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.314+0000 7f5ddffff700 1 -- 172.21.15.190:0/3981845167 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5de8013f00 con 0x7f5df80601e0 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.314+0000 7f5ddffff700 1 -- 172.21.15.190:0/3981845167 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5de8014d90 con 0x7f5df80601e0 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.314+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df8107080 con 0x7f5df80601e0 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.314+0000 7f5ddffff700 1 -- 172.21.15.190:0/3981845167 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5de80263a0 con 0x7f5df80601e0 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.314+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5df8045ad0 con 0x7f5df80601e0 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.314+0000 7f5ddffff700 1 --2- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5de002ff60 0x7f5de0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.315+0000 7f5ddffff700 1 -- 172.21.15.190:0/3981845167 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5de80413c0 con 0x7f5df80601e0 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.315+0000 7f5df6ffd700 1 --2- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5de002ff60 0x7f5de0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:35.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.315+0000 7f5df6ffd700 1 --2- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5de002ff60 0x7f5de0032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f5dec0096a0 tx=0x7f5dec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:35.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.319+0000 7f5ddffff700 1 -- 172.21.15.190:0/3981845167 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5de80126f0 con 0x7f5df80601e0 2024-01-16T17:11:35.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.525+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5df8107f30 con 0x7f5df80601e0 2024-01-16T17:11:35.508 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.527+0000 7f5ddffff700 1 -- 172.21.15.190:0/3981845167 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5de8012330 con 0x7f5df80601e0 2024-01-16T17:11:35.509 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:35.509 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:11:35.510 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5de002ff60 msgr2=0x7f5de0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:35.510 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 --2- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5de002ff60 0x7f5de0032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f5dec0096a0 tx=0x7f5dec006b40).stop 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 msgr2=0x7f5df8106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 --2- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 0x7f5df8106500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f5de8013510 tx=0x7f5de800b040).stop 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 shutdown_connections 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 --2- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5de002ff60 0x7f5de0032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 --2- 172.21.15.190:0/3981845167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5df80601e0 0x7f5df8106500 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 >> 172.21.15.190:0/3981845167 conn(0x7f5df80f09a0 msgr2=0x7f5df80f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 shutdown_connections 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:35.528+0000 7f5dfcae2700 1 -- 172.21.15.190:0/3981845167 wait complete. 2024-01-16T17:11:35.511 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:11:35.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:33.788340+0000 mgr.smithi154.sllivz (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:36.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:35.527972+0000 mon.smithi154 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.190:0/3981845167' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:11:36.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:36.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:37.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:35.788605+0000 mgr.smithi154.sllivz (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:38.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 -- 172.21.15.190:0/3624035143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 msgr2=0x7f78e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:38.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 --2- 172.21.15.190:0/3624035143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 0x7f78e40f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f78d4009fb0 tx=0x7f78d4009b30).stop 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 -- 172.21.15.190:0/3624035143 shutdown_connections 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 --2- 172.21.15.190:0/3624035143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 0x7f78e40f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 -- 172.21.15.190:0/3624035143 >> 172.21.15.190:0/3624035143 conn(0x7f78e40f0b70 msgr2=0x7f78e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 -- 172.21.15.190:0/3624035143 shutdown_connections 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 -- 172.21.15.190:0/3624035143 wait complete. 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 Processor -- start 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 -- start start 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 0x7f78e40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.070+0000 7f78eb725700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e40fb820 con 0x7f78e40f5410 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78e94c1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 0x7f78e40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78e94c1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 0x7f78e40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:42142/0 (socket says 172.21.15.190:42142) 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78e94c1700 1 -- 172.21.15.190:0/1729528461 learned_addr learned my addr 172.21.15.190:0/1729528461 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78e94c1700 1 -- 172.21.15.190:0/1729528461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d4005040 con 0x7f78e40f5410 2024-01-16T17:11:38.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78e94c1700 1 --2- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 0x7f78e40fd3a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f78d4014000 tx=0x7f78d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:38.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78d9ffb700 1 -- 172.21.15.190:0/1729528461 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d40095f0 con 0x7f78e40f5410 2024-01-16T17:11:38.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78d9ffb700 1 -- 172.21.15.190:0/1729528461 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f78d4007e70 con 0x7f78e40f5410 2024-01-16T17:11:38.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e40fb9c0 con 0x7f78e40f5410 2024-01-16T17:11:38.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78d9ffb700 1 -- 172.21.15.190:0/1729528461 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78d4014ea0 con 0x7f78e40f5410 2024-01-16T17:11:38.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.071+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e40fbe50 con 0x7f78e40f5410 2024-01-16T17:11:38.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.072+0000 7f78d9ffb700 1 -- 172.21.15.190:0/1729528461 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78d4025070 con 0x7f78e40f5410 2024-01-16T17:11:38.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.072+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e4059100 con 0x7f78e40f5410 2024-01-16T17:11:38.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.072+0000 7f78d9ffb700 1 --2- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f78d002ff60 0x7f78d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:38.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.072+0000 7f78d9ffb700 1 -- 172.21.15.190:0/1729528461 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f78d40409f0 con 0x7f78e40f5410 2024-01-16T17:11:38.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.073+0000 7f78e8cc0700 1 --2- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f78d002ff60 0x7f78d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:38.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.073+0000 7f78e8cc0700 1 --2- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f78d002ff60 0x7f78d0032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f78e00096a0 tx=0x7f78e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:38.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.077+0000 7f78d9ffb700 1 -- 172.21.15.190:0/1729528461 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f78d400f4e0 con 0x7f78e40f5410 2024-01-16T17:11:38.266 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.285+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78e40fae60 con 0x7f78e40f5410 2024-01-16T17:11:38.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.286+0000 7f78d9ffb700 1 -- 172.21.15.190:0/1729528461 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f78d400f300 con 0x7f78e40f5410 2024-01-16T17:11:38.268 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:38.268 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f78d002ff60 msgr2=0x7f78d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 --2- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f78d002ff60 0x7f78d0032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f78e00096a0 tx=0x7f78e0006b40).stop 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 msgr2=0x7f78e40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 --2- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 0x7f78e40fd3a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f78d4014000 tx=0x7f78d400b040).stop 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 shutdown_connections 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 --2- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f78d002ff60 0x7f78d0032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 --2- 172.21.15.190:0/1729528461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78e40f5410 0x7f78e40fd3a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 >> 172.21.15.190:0/1729528461 conn(0x7f78e40f0b70 msgr2=0x7f78e405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 shutdown_connections 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:38.287+0000 7f78eb725700 1 -- 172.21.15.190:0/1729528461 wait complete. 2024-01-16T17:11:38.270 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:11:38.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:38 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:38.287461+0000 mon.smithi154 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.190:0/1729528461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:11:39.681 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:39.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:39.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:37.788857+0000 mgr.smithi154.sllivz (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:40.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.868+0000 7ffaa134c700 1 -- 172.21.15.190:0/696935901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 msgr2=0x7ffa9c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:40.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.868+0000 7ffaa134c700 1 --2- 172.21.15.190:0/696935901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 0x7ffa9c0f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ffa90009fb0 tx=0x7ffa90009b30).stop 2024-01-16T17:11:40.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.868+0000 7ffaa134c700 1 -- 172.21.15.190:0/696935901 shutdown_connections 2024-01-16T17:11:40.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.868+0000 7ffaa134c700 1 --2- 172.21.15.190:0/696935901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 0x7ffa9c0f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:40.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.868+0000 7ffaa134c700 1 -- 172.21.15.190:0/696935901 >> 172.21.15.190:0/696935901 conn(0x7ffa9c0f1c70 msgr2=0x7ffa9c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.868+0000 7ffaa134c700 1 -- 172.21.15.190:0/696935901 shutdown_connections 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.868+0000 7ffaa134c700 1 -- 172.21.15.190:0/696935901 wait complete. 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.869+0000 7ffaa134c700 1 Processor -- start 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.869+0000 7ffaa134c700 1 -- start start 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.869+0000 7ffaa134c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 0x7ffa9c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.869+0000 7ffaa134c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c106c20 con 0x7ffa9c0f6510 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.869+0000 7ffa9affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 0x7ffa9c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.869+0000 7ffa9affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 0x7ffa9c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:42158/0 (socket says 172.21.15.190:42158) 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.869+0000 7ffa9affd700 1 -- 172.21.15.190:0/1053896609 learned_addr learned my addr 172.21.15.190:0/1053896609 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.870+0000 7ffa9affd700 1 -- 172.21.15.190:0/1053896609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa90005040 con 0x7ffa9c0f6510 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.870+0000 7ffa9affd700 1 --2- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 0x7ffa9c106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ffa90013510 tx=0x7ffa9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.870+0000 7ffa837fe700 1 -- 172.21.15.190:0/1053896609 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa90009320 con 0x7ffa9c0f6510 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.870+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c106e20 con 0x7ffa9c0f6510 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.870+0000 7ffa837fe700 1 -- 172.21.15.190:0/1053896609 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa90013f00 con 0x7ffa9c0f6510 2024-01-16T17:11:40.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.870+0000 7ffa837fe700 1 -- 172.21.15.190:0/1053896609 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa90014d90 con 0x7ffa9c0f6510 2024-01-16T17:11:40.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.870+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c1072b0 con 0x7ffa9c0f6510 2024-01-16T17:11:40.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.871+0000 7ffa837fe700 1 -- 172.21.15.190:0/1053896609 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa90014420 con 0x7ffa9c0f6510 2024-01-16T17:11:40.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.871+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c045ad0 con 0x7ffa9c0f6510 2024-01-16T17:11:40.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.871+0000 7ffa837fe700 1 --2- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffa8402ff60 0x7ffa84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:40.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.871+0000 7ffa837fe700 1 -- 172.21.15.190:0/1053896609 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa90041000 con 0x7ffa9c0f6510 2024-01-16T17:11:40.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.871+0000 7ffa9a7fc700 1 --2- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffa8402ff60 0x7ffa84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:40.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.871+0000 7ffa9a7fc700 1 --2- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffa8402ff60 0x7ffa84032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ffa8c0096a0 tx=0x7ffa8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:40.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:40.876+0000 7ffa837fe700 1 -- 172.21.15.190:0/1053896609 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa900095f0 con 0x7ffa9c0f6510 2024-01-16T17:11:41.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.079+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa9c0fbf60 con 0x7ffa9c0f6510 2024-01-16T17:11:41.062 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.080+0000 7ffa837fe700 1 -- 172.21.15.190:0/1053896609 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffa9001db70 con 0x7ffa9c0f6510 2024-01-16T17:11:41.062 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:41.063 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:11:41.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffa8402ff60 msgr2=0x7ffa84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:41.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 --2- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffa8402ff60 0x7ffa84032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7ffa8c0096a0 tx=0x7ffa8c006b40).stop 2024-01-16T17:11:41.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 msgr2=0x7ffa9c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:41.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 --2- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 0x7ffa9c106730 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ffa90013510 tx=0x7ffa9000b040).stop 2024-01-16T17:11:41.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 shutdown_connections 2024-01-16T17:11:41.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 --2- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffa8402ff60 0x7ffa84032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:41.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 --2- 172.21.15.190:0/1053896609 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa9c0f6510 0x7ffa9c106730 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:41.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 >> 172.21.15.190:0/1053896609 conn(0x7ffa9c0f1c70 msgr2=0x7ffa9c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:41.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 shutdown_connections 2024-01-16T17:11:41.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:41.082+0000 7ffaa134c700 1 -- 172.21.15.190:0/1053896609 wait complete. 2024-01-16T17:11:41.065 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:11:41.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:39.789165+0000 mgr.smithi154.sllivz (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:41.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:41.082001+0000 mon.smithi154 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.190:0/1053896609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:11:42.454 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:42.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:43.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 -- 172.21.15.190:0/594235831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 msgr2=0x7f2f380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 --2- 172.21.15.190:0/594235831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 0x7f2f380f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f2f28009fb0 tx=0x7f2f28009b30).stop 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 -- 172.21.15.190:0/594235831 shutdown_connections 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 --2- 172.21.15.190:0/594235831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 0x7f2f380f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 -- 172.21.15.190:0/594235831 >> 172.21.15.190:0/594235831 conn(0x7f2f380f1c70 msgr2=0x7f2f380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 -- 172.21.15.190:0/594235831 shutdown_connections 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 -- 172.21.15.190:0/594235831 wait complete. 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 Processor -- start 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.595+0000 7f2f400ed700 1 -- start start 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f400ed700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 0x7f2f38106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f400ed700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f38106c20 con 0x7f2f380f6510 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f3de89700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 0x7f2f38106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f3de89700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 0x7f2f38106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:37080/0 (socket says 172.21.15.190:37080) 2024-01-16T17:11:43.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f3de89700 1 -- 172.21.15.190:0/2118567991 learned_addr learned my addr 172.21.15.190:0/2118567991 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:43.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f3de89700 1 -- 172.21.15.190:0/2118567991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f28005040 con 0x7f2f380f6510 2024-01-16T17:11:43.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f3de89700 1 --2- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 0x7f2f38106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2f28014000 tx=0x7f2f2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:43.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f2e7fc700 1 -- 172.21.15.190:0/2118567991 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f280095f0 con 0x7f2f380f6510 2024-01-16T17:11:43.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f38106e20 con 0x7f2f380f6510 2024-01-16T17:11:43.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f2e7fc700 1 -- 172.21.15.190:0/2118567991 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f28007e70 con 0x7f2f380f6510 2024-01-16T17:11:43.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f2e7fc700 1 -- 172.21.15.190:0/2118567991 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f28014ea0 con 0x7f2f380f6510 2024-01-16T17:11:43.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.596+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f381072b0 con 0x7f2f380f6510 2024-01-16T17:11:43.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.597+0000 7f2f2e7fc700 1 -- 172.21.15.190:0/2118567991 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f28025070 con 0x7f2f380f6510 2024-01-16T17:11:43.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.597+0000 7f2f2e7fc700 1 --2- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2f2402ff60 0x7f2f24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:43.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.597+0000 7f2f2e7fc700 1 -- 172.21.15.190:0/2118567991 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f280409c0 con 0x7f2f380f6510 2024-01-16T17:11:43.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.597+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f38045ad0 con 0x7f2f380f6510 2024-01-16T17:11:43.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.597+0000 7f2f3d688700 1 --2- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2f2402ff60 0x7f2f24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:43.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.598+0000 7f2f3d688700 1 --2- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2f2402ff60 0x7f2f24032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f2f340096a0 tx=0x7f2f34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:43.583 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.602+0000 7f2f2e7fc700 1 -- 172.21.15.190:0/2118567991 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f28026350 con 0x7f2f380f6510 2024-01-16T17:11:43.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:41.789506+0000 mgr.smithi154.sllivz (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:43.788 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.807+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f380fc050 con 0x7f2f380f6510 2024-01-16T17:11:43.789 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.808+0000 7f2f2e7fc700 1 -- 172.21.15.190:0/2118567991 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2f28012330 con 0x7f2f380f6510 2024-01-16T17:11:43.789 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:43.789 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2f2402ff60 msgr2=0x7f2f24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 --2- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2f2402ff60 0x7f2f24032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f2f340096a0 tx=0x7f2f34006b40).stop 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 msgr2=0x7f2f38106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 --2- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 0x7f2f38106730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f2f28014000 tx=0x7f2f2800b040).stop 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 shutdown_connections 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 --2- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2f2402ff60 0x7f2f24032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 --2- 172.21.15.190:0/2118567991 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f380f6510 0x7f2f38106730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 >> 172.21.15.190:0/2118567991 conn(0x7f2f380f1c70 msgr2=0x7f2f380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 shutdown_connections 2024-01-16T17:11:43.791 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:43.810+0000 7f2f400ed700 1 -- 172.21.15.190:0/2118567991 wait complete. 2024-01-16T17:11:43.792 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:11:44.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:44 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:43.808801+0000 mon.smithi154 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.190:0/2118567991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:11:45.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:45.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:45.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:43.789845+0000 mgr.smithi154.sllivz (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:47.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.267+0000 7f34f145a700 1 -- 172.21.15.190:0/3783954243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 msgr2=0x7f34ec0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:47.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.267+0000 7f34f145a700 1 --2- 172.21.15.190:0/3783954243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 0x7f34ec0f9150 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f34e0009fb0 tx=0x7f34e0009b30).stop 2024-01-16T17:11:47.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.267+0000 7f34f145a700 1 -- 172.21.15.190:0/3783954243 shutdown_connections 2024-01-16T17:11:47.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.267+0000 7f34f145a700 1 --2- 172.21.15.190:0/3783954243 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 0x7f34ec0f9150 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.267+0000 7f34f145a700 1 -- 172.21.15.190:0/3783954243 >> 172.21.15.190:0/3783954243 conn(0x7f34ec0f0b70 msgr2=0x7f34ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.267+0000 7f34f145a700 1 -- 172.21.15.190:0/3783954243 shutdown_connections 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.267+0000 7f34f145a700 1 -- 172.21.15.190:0/3783954243 wait complete. 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.268+0000 7f34f145a700 1 Processor -- start 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.268+0000 7f34f145a700 1 -- start start 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.268+0000 7f34f145a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 0x7f34ec0f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.268+0000 7f34f145a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec0f5050 con 0x7f34ec060220 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.268+0000 7f34eaffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 0x7f34ec0f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.268+0000 7f34eaffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 0x7f34ec0f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:37112/0 (socket says 172.21.15.190:37112) 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.268+0000 7f34eaffd700 1 -- 172.21.15.190:0/3525351059 learned_addr learned my addr 172.21.15.190:0/3525351059 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:47.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.269+0000 7f34eaffd700 1 -- 172.21.15.190:0/3525351059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e0005040 con 0x7f34ec060220 2024-01-16T17:11:47.252 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.269+0000 7f34eaffd700 1 --2- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 0x7f34ec0f6bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f34e0009830 tx=0x7f34e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:47.252 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.269+0000 7f34d37fe700 1 -- 172.21.15.190:0/3525351059 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e0009320 con 0x7f34ec060220 2024-01-16T17:11:47.252 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.269+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ec0f51f0 con 0x7f34ec060220 2024-01-16T17:11:47.252 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.269+0000 7f34d37fe700 1 -- 172.21.15.190:0/3525351059 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34e00136c0 con 0x7f34ec060220 2024-01-16T17:11:47.252 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.269+0000 7f34d37fe700 1 -- 172.21.15.190:0/3525351059 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34e001ee90 con 0x7f34ec060220 2024-01-16T17:11:47.252 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.269+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ec0f5680 con 0x7f34ec060220 2024-01-16T17:11:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.270+0000 7f34d37fe700 1 -- 172.21.15.190:0/3525351059 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34e001e420 con 0x7f34ec060220 2024-01-16T17:11:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.270+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ec059100 con 0x7f34ec060220 2024-01-16T17:11:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.270+0000 7f34d37fe700 1 --2- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f34d402ff10 0x7f34d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.270+0000 7f34d37fe700 1 -- 172.21.15.190:0/3525351059 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34e0040fa0 con 0x7f34ec060220 2024-01-16T17:11:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.270+0000 7f34ea7fc700 1 --2- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f34d402ff10 0x7f34d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.270+0000 7f34ea7fc700 1 --2- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f34d402ff10 0x7f34d40323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f34dc0096a0 tx=0x7f34dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:47.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.274+0000 7f34d37fe700 1 -- 172.21.15.190:0/3525351059 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34e00095f0 con 0x7f34ec060220 2024-01-16T17:11:47.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.482+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34ec0fbf50 con 0x7f34ec060220 2024-01-16T17:11:47.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.483+0000 7f34d37fe700 1 -- 172.21.15.190:0/3525351059 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f34e002c2d0 con 0x7f34ec060220 2024-01-16T17:11:47.464 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:47.464 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:11:47.465 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f34d402ff10 msgr2=0x7f34d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 --2- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f34d402ff10 0x7f34d40323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f34dc0096a0 tx=0x7f34dc006b40).stop 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 msgr2=0x7f34ec0f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 --2- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 0x7f34ec0f6bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f34e0009830 tx=0x7f34e000b040).stop 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 shutdown_connections 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 --2- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f34d402ff10 0x7f34d40323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 --2- 172.21.15.190:0/3525351059 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34ec060220 0x7f34ec0f6bd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 >> 172.21.15.190:0/3525351059 conn(0x7f34ec0f0b70 msgr2=0x7f34ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 shutdown_connections 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:47.484+0000 7f34f145a700 1 -- 172.21.15.190:0/3525351059 wait complete. 2024-01-16T17:11:47.466 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:11:47.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:45.790100+0000 mgr.smithi154.sllivz (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:48.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:48 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:47.483615+0000 mon.smithi154 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.190:0/3525351059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:11:48.841 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:48.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:49.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:47.790409+0000 mgr.smithi154.sllivz (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.961+0000 7f21f9779700 1 -- 172.21.15.190:0/2447453713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 msgr2=0x7f21f40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.961+0000 7f21f9779700 1 --2- 172.21.15.190:0/2447453713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 0x7f21f40f90f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f21dc009fb0 tx=0x7f21dc009b30).stop 2024-01-16T17:11:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.961+0000 7f21f9779700 1 -- 172.21.15.190:0/2447453713 shutdown_connections 2024-01-16T17:11:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.961+0000 7f21f9779700 1 --2- 172.21.15.190:0/2447453713 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 0x7f21f40f90f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.961+0000 7f21f9779700 1 -- 172.21.15.190:0/2447453713 >> 172.21.15.190:0/2447453713 conn(0x7f21f40f0b70 msgr2=0x7f21f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.961+0000 7f21f9779700 1 -- 172.21.15.190:0/2447453713 shutdown_connections 2024-01-16T17:11:49.943 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.961+0000 7f21f9779700 1 -- 172.21.15.190:0/2447453713 wait complete. 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f9779700 1 Processor -- start 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f9779700 1 -- start start 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f9779700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 0x7f21f41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f9779700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f4106bd0 con 0x7f21f4060280 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 0x7f21f41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 0x7f21f41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:37128/0 (socket says 172.21.15.190:37128) 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f2ffd700 1 -- 172.21.15.190:0/3940983507 learned_addr learned my addr 172.21.15.190:0/3940983507 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f2ffd700 1 -- 172.21.15.190:0/3940983507 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21dc005040 con 0x7f21f4060280 2024-01-16T17:11:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.962+0000 7f21f2ffd700 1 --2- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 0x7f21f41066e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f21dc013510 tx=0x7f21dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:49.945 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.963+0000 7f21eb7fe700 1 -- 172.21.15.190:0/3940983507 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21dc009320 con 0x7f21f4060280 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.963+0000 7f21eb7fe700 1 -- 172.21.15.190:0/3940983507 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f21dc013f00 con 0x7f21f4060280 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.963+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f4106dd0 con 0x7f21f4060280 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.963+0000 7f21eb7fe700 1 -- 172.21.15.190:0/3940983507 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f21dc014dc0 con 0x7f21f4060280 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.963+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f4107260 con 0x7f21f4060280 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.963+0000 7f21eb7fe700 1 -- 172.21.15.190:0/3940983507 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21dc014420 con 0x7f21f4060280 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.963+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f4045ad0 con 0x7f21f4060280 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.964+0000 7f21eb7fe700 1 --2- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f21e002ff60 0x7f21e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.964+0000 7f21eb7fe700 1 -- 172.21.15.190:0/3940983507 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f21dc041110 con 0x7f21f4060280 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.964+0000 7f21f27fc700 1 --2- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f21e002ff60 0x7f21e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:49.946 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.964+0000 7f21f27fc700 1 --2- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f21e002ff60 0x7f21e0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f21e40096a0 tx=0x7f21e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:49.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:49.968+0000 7f21eb7fe700 1 -- 172.21.15.190:0/3940983507 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21dc0095f0 con 0x7f21f4060280 2024-01-16T17:11:50.153 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.172+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21f4108110 con 0x7f21f4060280 2024-01-16T17:11:50.154 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.173+0000 7f21eb7fe700 1 -- 172.21.15.190:0/3940983507 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f21dc01cb10 con 0x7f21f4060280 2024-01-16T17:11:50.154 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:50.154 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f21e002ff60 msgr2=0x7f21e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 --2- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f21e002ff60 0x7f21e0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f21e40096a0 tx=0x7f21e4006b40).stop 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 msgr2=0x7f21f41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 --2- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 0x7f21f41066e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f21dc013510 tx=0x7f21dc00b040).stop 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 shutdown_connections 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 --2- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f21e002ff60 0x7f21e0032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 --2- 172.21.15.190:0/3940983507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21f4060280 0x7f21f41066e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 >> 172.21.15.190:0/3940983507 conn(0x7f21f40f0b70 msgr2=0x7f21f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:50.156 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.174+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 shutdown_connections 2024-01-16T17:11:50.157 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:50.175+0000 7f21f9779700 1 -- 172.21.15.190:0/3940983507 wait complete. 2024-01-16T17:11:50.157 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:11:50.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:50 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:50.173812+0000 mon.smithi154 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.190:0/3940983507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:11:51.568 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:51.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:51.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:49.790726+0000 mgr.smithi154.sllivz (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:53.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:51.791099+0000 mgr.smithi154.sllivz (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:55.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:53.791427+0000 mgr.smithi154.sllivz (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:56.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.595+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/1123333481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 msgr2=0x7f8f780f6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.595+0000 7f8f7cc6e700 1 --2- 172.21.15.190:0/1123333481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 0x7f8f780f6ee0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f8f68009fb0 tx=0x7f8f68009b30).stop 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.595+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/1123333481 shutdown_connections 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.595+0000 7f8f7cc6e700 1 --2- 172.21.15.190:0/1123333481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 0x7f8f780f6ee0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.595+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/1123333481 >> 172.21.15.190:0/1123333481 conn(0x7f8f780f0b70 msgr2=0x7f8f780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.595+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/1123333481 shutdown_connections 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.595+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/1123333481 wait complete. 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.596+0000 7f8f7cc6e700 1 Processor -- start 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.596+0000 7f8f7cc6e700 1 -- start start 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.596+0000 7f8f7cc6e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 0x7f8f7810cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.596+0000 7f8f7cc6e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7810d220 con 0x7f8f78060280 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f7659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 0x7f8f7810cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f7659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 0x7f8f7810cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33190/0 (socket says 172.21.15.190:33190) 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f7659c700 1 -- 172.21.15.190:0/713528484 learned_addr learned my addr 172.21.15.190:0/713528484 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:11:56.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f7659c700 1 -- 172.21.15.190:0/713528484 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f68005040 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f7659c700 1 --2- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 0x7f8f7810cd30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8f68013510 tx=0x7f8f6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f66ffd700 1 -- 172.21.15.190:0/713528484 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f68009320 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f66ffd700 1 -- 172.21.15.190:0/713528484 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f68013f00 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f66ffd700 1 -- 172.21.15.190:0/713528484 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f68014dc0 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7810d420 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.597+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7810e030 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.598+0000 7f8f66ffd700 1 -- 172.21.15.190:0/713528484 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f680263a0 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.598+0000 7f8f66ffd700 1 --2- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f6002ffb0 0x7f8f60032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.598+0000 7f8f66ffd700 1 -- 172.21.15.190:0/713528484 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f680413b0 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.598+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7810d8b0 con 0x7f8f78060280 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.598+0000 7f8f75d9b700 1 --2- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f6002ffb0 0x7f8f60032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.599+0000 7f8f75d9b700 1 --2- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f6002ffb0 0x7f8f60032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8f6c0096a0 tx=0x7f8f6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:11:56.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.603+0000 7f8f66ffd700 1 -- 172.21.15.190:0/713528484 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f7810d8b0 con 0x7f8f78060280 2024-01-16T17:11:56.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:55.925471+0000 mon.smithi154 (mon.0) 127 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:11:56.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:55.927538+0000 mon.smithi154 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:11:56.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.809+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f7810d8b0 con 0x7f8f78060280 2024-01-16T17:11:56.790 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.809+0000 7f8f66ffd700 1 -- 172.21.15.190:0/713528484 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8f7810d8b0 con 0x7f8f78060280 2024-01-16T17:11:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:11:56.791 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:11:56.792 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f6002ffb0 msgr2=0x7f8f60032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 --2- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f6002ffb0 0x7f8f60032470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f8f6c0096a0 tx=0x7f8f6c006b40).stop 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 msgr2=0x7f8f7810cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 --2- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 0x7f8f7810cd30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f8f68013510 tx=0x7f8f6800b040).stop 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 shutdown_connections 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 --2- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f6002ffb0 0x7f8f60032470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 --2- 172.21.15.190:0/713528484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f78060280 0x7f8f7810cd30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 >> 172.21.15.190:0/713528484 conn(0x7f8f780f0b70 msgr2=0x7f8f780f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 shutdown_connections 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:11:56.811+0000 7f8f7cc6e700 1 -- 172.21.15.190:0/713528484 wait complete. 2024-01-16T17:11:56.793 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:11:57.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:55.791640+0000 mgr.smithi154.sllivz (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:11:57.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:11:56.810451+0000 mon.smithi154 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.190:0/713528484' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:11:59.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:11:59.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:11:59.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:11:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:57.791959+0000 mgr.smithi154.sllivz (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:00.779 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.795+0000 7f6b97f43700 1 -- 172.21.15.190:0/775980475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 msgr2=0x7f6b900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:00.779 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.795+0000 7f6b97f43700 1 --2- 172.21.15.190:0/775980475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 0x7f6b900f6930 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f6b80009fb0 tx=0x7f6b80009b30).stop 2024-01-16T17:12:00.779 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.795+0000 7f6b97f43700 1 -- 172.21.15.190:0/775980475 shutdown_connections 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.795+0000 7f6b97f43700 1 --2- 172.21.15.190:0/775980475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 0x7f6b900f6930 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.795+0000 7f6b97f43700 1 -- 172.21.15.190:0/775980475 >> 172.21.15.190:0/775980475 conn(0x7f6b900f1c70 msgr2=0x7f6b900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.796+0000 7f6b97f43700 1 -- 172.21.15.190:0/775980475 shutdown_connections 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.796+0000 7f6b97f43700 1 -- 172.21.15.190:0/775980475 wait complete. 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.796+0000 7f6b97f43700 1 Processor -- start 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.796+0000 7f6b97f43700 1 -- start start 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.796+0000 7f6b97f43700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 0x7f6b90106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.796+0000 7f6b97f43700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90106c20 con 0x7f6b900f6510 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b95cdf700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 0x7f6b90106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b95cdf700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 0x7f6b90106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33202/0 (socket says 172.21.15.190:33202) 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b95cdf700 1 -- 172.21.15.190:0/538922660 learned_addr learned my addr 172.21.15.190:0/538922660 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b95cdf700 1 -- 172.21.15.190:0/538922660 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b80005040 con 0x7f6b900f6510 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b95cdf700 1 --2- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 0x7f6b90106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6b80013510 tx=0x7f6b8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:00.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b867fc700 1 -- 172.21.15.190:0/538922660 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b80009320 con 0x7f6b900f6510 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b90106e20 con 0x7f6b900f6510 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b867fc700 1 -- 172.21.15.190:0/538922660 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b80013f00 con 0x7f6b900f6510 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b867fc700 1 -- 172.21.15.190:0/538922660 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b80014d90 con 0x7f6b900f6510 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.797+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b901072b0 con 0x7f6b900f6510 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.798+0000 7f6b867fc700 1 -- 172.21.15.190:0/538922660 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b800095f0 con 0x7f6b900f6510 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.798+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b90045ad0 con 0x7f6b900f6510 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.798+0000 7f6b867fc700 1 --2- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b7c02ff60 0x7f6b7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.798+0000 7f6b867fc700 1 -- 172.21.15.190:0/538922660 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b800410c0 con 0x7f6b900f6510 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.798+0000 7f6b954de700 1 --2- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b7c02ff60 0x7f6b7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.799+0000 7f6b954de700 1 --2- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b7c02ff60 0x7f6b7c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f6b8c0096a0 tx=0x7f6b8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:00.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:00.803+0000 7f6b867fc700 1 -- 172.21.15.190:0/538922660 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b80014420 con 0x7f6b900f6510 2024-01-16T17:12:00.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.007+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b900fbf60 con 0x7f6b900f6510 2024-01-16T17:12:00.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.008+0000 7f6b867fc700 1 -- 172.21.15.190:0/538922660 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6b80010230 con 0x7f6b900f6510 2024-01-16T17:12:00.989 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:00.989 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b7c02ff60 msgr2=0x7f6b7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 --2- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b7c02ff60 0x7f6b7c032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f6b8c0096a0 tx=0x7f6b8c006b40).stop 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 msgr2=0x7f6b90106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 --2- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 0x7f6b90106730 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f6b80013510 tx=0x7f6b8000b040).stop 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 shutdown_connections 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 --2- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b7c02ff60 0x7f6b7c032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 --2- 172.21.15.190:0/538922660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b900f6510 0x7f6b90106730 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 >> 172.21.15.190:0/538922660 conn(0x7f6b900f1c70 msgr2=0x7f6b900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 shutdown_connections 2024-01-16T17:12:00.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:01.010+0000 7f6b97f43700 1 -- 172.21.15.190:0/538922660 wait complete. 2024-01-16T17:12:00.992 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:01.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:11:59.792273+0000 mgr.smithi154.sllivz (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:01.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:01.008941+0000 mon.smithi154 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.190:0/538922660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:02.372 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:02.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:03.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.467+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/3774928437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 msgr2=0x7fbf580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:03.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.467+0000 7fbf5f9d5700 1 --2- 172.21.15.190:0/3774928437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 0x7fbf580f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fbf4c009fb0 tx=0x7fbf4c009b30).stop 2024-01-16T17:12:03.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.467+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/3774928437 shutdown_connections 2024-01-16T17:12:03.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.467+0000 7fbf5f9d5700 1 --2- 172.21.15.190:0/3774928437 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 0x7fbf580f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:03.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.467+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/3774928437 >> 172.21.15.190:0/3774928437 conn(0x7fbf580f0b70 msgr2=0x7fbf580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:03.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.467+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/3774928437 shutdown_connections 2024-01-16T17:12:03.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.467+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/3774928437 wait complete. 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5f9d5700 1 Processor -- start 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5f9d5700 1 -- start start 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5f9d5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 0x7fbf58106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5f9d5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf58106c20 con 0x7fbf580f5410 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5d771700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 0x7fbf58106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5d771700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 0x7fbf58106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52858/0 (socket says 172.21.15.190:52858) 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5d771700 1 -- 172.21.15.190:0/1890122561 learned_addr learned my addr 172.21.15.190:0/1890122561 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5d771700 1 -- 172.21.15.190:0/1890122561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c005040 con 0x7fbf580f5410 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5d771700 1 --2- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 0x7fbf58106730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fbf4c013510 tx=0x7fbf4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf49ffb700 1 -- 172.21.15.190:0/1890122561 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf4c009320 con 0x7fbf580f5410 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf58106e20 con 0x7fbf580f5410 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.468+0000 7fbf49ffb700 1 -- 172.21.15.190:0/1890122561 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf4c013f00 con 0x7fbf580f5410 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.469+0000 7fbf49ffb700 1 -- 172.21.15.190:0/1890122561 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf4c014d90 con 0x7fbf580f5410 2024-01-16T17:12:03.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.469+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf581072b0 con 0x7fbf580f5410 2024-01-16T17:12:03.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.469+0000 7fbf49ffb700 1 -- 172.21.15.190:0/1890122561 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf4c0095f0 con 0x7fbf580f5410 2024-01-16T17:12:03.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.470+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf58045ad0 con 0x7fbf580f5410 2024-01-16T17:12:03.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.470+0000 7fbf49ffb700 1 --2- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbf4402ff60 0x7fbf44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:03.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.470+0000 7fbf49ffb700 1 -- 172.21.15.190:0/1890122561 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf4c0410a0 con 0x7fbf580f5410 2024-01-16T17:12:03.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.470+0000 7fbf5cf70700 1 --2- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbf4402ff60 0x7fbf44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:03.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.470+0000 7fbf5cf70700 1 --2- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbf4402ff60 0x7fbf44032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fbf540096a0 tx=0x7fbf54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:03.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.474+0000 7fbf49ffb700 1 -- 172.21.15.190:0/1890122561 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf4c01d7f0 con 0x7fbf580f5410 2024-01-16T17:12:03.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.682+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf580fae60 con 0x7fbf580f5410 2024-01-16T17:12:03.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.682+0000 7fbf49ffb700 1 -- 172.21.15.190:0/1890122561 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbf4c010230 con 0x7fbf580f5410 2024-01-16T17:12:03.663 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:03.663 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:03.665 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbf4402ff60 msgr2=0x7fbf44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:03.665 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 --2- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbf4402ff60 0x7fbf44032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fbf540096a0 tx=0x7fbf54006b40).stop 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 msgr2=0x7fbf58106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 --2- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 0x7fbf58106730 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fbf4c013510 tx=0x7fbf4c00b040).stop 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 shutdown_connections 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 --2- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbf4402ff60 0x7fbf44032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 --2- 172.21.15.190:0/1890122561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf580f5410 0x7fbf58106730 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 >> 172.21.15.190:0/1890122561 conn(0x7fbf580f0b70 msgr2=0x7fbf580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 shutdown_connections 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:03.684+0000 7fbf5f9d5700 1 -- 172.21.15.190:0/1890122561 wait complete. 2024-01-16T17:12:03.666 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:03.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:01.792673+0000 mgr.smithi154.sllivz (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:04.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:04 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:03.683220+0000 mon.smithi154 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.190:0/1890122561' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:05.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:05.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:05.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:03.793036+0000 mgr.smithi154.sllivz (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:06.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.226+0000 7f2166c6d700 1 -- 172.21.15.190:0/2149291348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 msgr2=0x7f21600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:06.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.226+0000 7f2166c6d700 1 --2- 172.21.15.190:0/2149291348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 0x7f21600f6930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f2158009960 tx=0x7f2158004d10).stop 2024-01-16T17:12:06.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.226+0000 7f2166c6d700 1 -- 172.21.15.190:0/2149291348 shutdown_connections 2024-01-16T17:12:06.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.226+0000 7f2166c6d700 1 --2- 172.21.15.190:0/2149291348 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 0x7f21600f6930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:06.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.226+0000 7f2166c6d700 1 -- 172.21.15.190:0/2149291348 >> 172.21.15.190:0/2149291348 conn(0x7f21600f1c70 msgr2=0x7f21600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:06.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.226+0000 7f2166c6d700 1 -- 172.21.15.190:0/2149291348 shutdown_connections 2024-01-16T17:12:06.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.226+0000 7f2166c6d700 1 -- 172.21.15.190:0/2149291348 wait complete. 2024-01-16T17:12:06.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.227+0000 7f2166c6d700 1 Processor -- start 2024-01-16T17:12:06.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.227+0000 7f2166c6d700 1 -- start start 2024-01-16T17:12:06.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.227+0000 7f2166c6d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 0x7f2160106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:06.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.227+0000 7f2166c6d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2160106c30 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.227+0000 7f2164a09700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 0x7f2160106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.227+0000 7f2164a09700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 0x7f2160106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52880/0 (socket says 172.21.15.190:52880) 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.227+0000 7f2164a09700 1 -- 172.21.15.190:0/2115972851 learned_addr learned my addr 172.21.15.190:0/2115972851 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.227+0000 7f2164a09700 1 -- 172.21.15.190:0/2115972851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2158009640 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.228+0000 7f2164a09700 1 --2- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 0x7f2160106740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2158004a10 tx=0x7f2158004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.228+0000 7f215d7fa700 1 -- 172.21.15.190:0/2115972851 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2158018070 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.228+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2160106e30 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.228+0000 7f215d7fa700 1 -- 172.21.15.190:0/2115972851 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2158007e70 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.228+0000 7f215d7fa700 1 -- 172.21.15.190:0/2115972851 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f215801d420 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.228+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21601072c0 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.229+0000 7f215d7fa700 1 -- 172.21.15.190:0/2115972851 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2158013630 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.229+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2160045ad0 con 0x7f21600f6510 2024-01-16T17:12:06.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.229+0000 7f215d7fa700 1 --2- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f214c02ff60 0x7f214c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:06.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.229+0000 7f215d7fa700 1 -- 172.21.15.190:0/2115972851 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2158041680 con 0x7f21600f6510 2024-01-16T17:12:06.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.229+0000 7f215ffff700 1 --2- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f214c02ff60 0x7f214c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:06.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.229+0000 7f215ffff700 1 --2- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f214c02ff60 0x7f214c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f21480096a0 tx=0x7f2148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:06.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.233+0000 7f215d7fa700 1 -- 172.21.15.190:0/2115972851 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f215801bd40 con 0x7f21600f6510 2024-01-16T17:12:06.420 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.440+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21600fbf60 con 0x7f21600f6510 2024-01-16T17:12:06.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.440+0000 7f215d7fa700 1 -- 172.21.15.190:0/2115972851 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f215800b870 con 0x7f21600f6510 2024-01-16T17:12:06.421 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:06.421 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f214c02ff60 msgr2=0x7f214c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 --2- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f214c02ff60 0x7f214c032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f21480096a0 tx=0x7f2148006b40).stop 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 msgr2=0x7f2160106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 --2- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 0x7f2160106740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2158004a10 tx=0x7f2158004d10).stop 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 shutdown_connections 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 --2- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f214c02ff60 0x7f214c032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 --2- 172.21.15.190:0/2115972851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21600f6510 0x7f2160106740 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 >> 172.21.15.190:0/2115972851 conn(0x7f21600f1c70 msgr2=0x7f21600f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 shutdown_connections 2024-01-16T17:12:06.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:06.442+0000 7f2166c6d700 1 -- 172.21.15.190:0/2115972851 wait complete. 2024-01-16T17:12:06.424 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:06.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:06.441378+0000 mon.smithi154 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.190:0/2115972851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:05.793420+0000 mgr.smithi154.sllivz (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:07.819 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:07.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:08.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.946+0000 7f1d2194c700 1 -- 172.21.15.190:0/867076581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 msgr2=0x7f1d1c0f6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:08.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.946+0000 7f1d2194c700 1 --2- 172.21.15.190:0/867076581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 0x7f1d1c0f6660 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f1d10009fb0 tx=0x7f1d10009b30).stop 2024-01-16T17:12:08.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.946+0000 7f1d2194c700 1 -- 172.21.15.190:0/867076581 shutdown_connections 2024-01-16T17:12:08.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.946+0000 7f1d2194c700 1 --2- 172.21.15.190:0/867076581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 0x7f1d1c0f6660 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:08.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.946+0000 7f1d2194c700 1 -- 172.21.15.190:0/867076581 >> 172.21.15.190:0/867076581 conn(0x7f1d1c0f19c0 msgr2=0x7f1d1c0f3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:08.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.946+0000 7f1d2194c700 1 -- 172.21.15.190:0/867076581 shutdown_connections 2024-01-16T17:12:08.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.947+0000 7f1d2194c700 1 -- 172.21.15.190:0/867076581 wait complete. 2024-01-16T17:12:08.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.947+0000 7f1d2194c700 1 Processor -- start 2024-01-16T17:12:08.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.947+0000 7f1d2194c700 1 -- start start 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d2194c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 0x7f1d1c1064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d2194c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c1069c0 con 0x7f1d1c0f6240 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d2094a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 0x7f1d1c1064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d2094a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 0x7f1d1c1064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52900/0 (socket says 172.21.15.190:52900) 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d2094a700 1 -- 172.21.15.190:0/2238997135 learned_addr learned my addr 172.21.15.190:0/2238997135 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d2094a700 1 -- 172.21.15.190:0/2238997135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d10005040 con 0x7f1d1c0f6240 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d2094a700 1 --2- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 0x7f1d1c1064d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1d10014000 tx=0x7f1d1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d197fa700 1 -- 172.21.15.190:0/2238997135 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d100095f0 con 0x7f1d1c0f6240 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d197fa700 1 -- 172.21.15.190:0/2238997135 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d10007e70 con 0x7f1d1c0f6240 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.948+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1c106bc0 con 0x7f1d1c0f6240 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.949+0000 7f1d197fa700 1 -- 172.21.15.190:0/2238997135 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d10014ea0 con 0x7f1d1c0f6240 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.949+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1c107050 con 0x7f1d1c0f6240 2024-01-16T17:12:08.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.949+0000 7f1d197fa700 1 -- 172.21.15.190:0/2238997135 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d10025070 con 0x7f1d1c0f6240 2024-01-16T17:12:08.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.949+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1c045ad0 con 0x7f1d1c0f6240 2024-01-16T17:12:08.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.949+0000 7f1d197fa700 1 --2- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1d0c02ffb0 0x7f1d0c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:08.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.949+0000 7f1d197fa700 1 -- 172.21.15.190:0/2238997135 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d10040a30 con 0x7f1d1c0f6240 2024-01-16T17:12:08.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.950+0000 7f1d1bfff700 1 --2- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1d0c02ffb0 0x7f1d0c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:08.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.950+0000 7f1d1bfff700 1 --2- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1d0c02ffb0 0x7f1d0c032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1d080096a0 tx=0x7f1d08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:08.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:08.954+0000 7f1d197fa700 1 -- 172.21.15.190:0/2238997135 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d100079c0 con 0x7f1d1c0f6240 2024-01-16T17:12:09.141 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.159+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d1c0fbd00 con 0x7f1d1c0f6240 2024-01-16T17:12:09.141 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.160+0000 7f1d197fa700 1 -- 172.21.15.190:0/2238997135 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1d1000f300 con 0x7f1d1c0f6240 2024-01-16T17:12:09.142 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:09.142 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:09.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.161+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1d0c02ffb0 msgr2=0x7f1d0c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:09.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.161+0000 7f1d2194c700 1 --2- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1d0c02ffb0 0x7f1d0c032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f1d080096a0 tx=0x7f1d08006b40).stop 2024-01-16T17:12:09.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.161+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 msgr2=0x7f1d1c1064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:09.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.161+0000 7f1d2194c700 1 --2- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 0x7f1d1c1064d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f1d10014000 tx=0x7f1d1000b040).stop 2024-01-16T17:12:09.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.161+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 shutdown_connections 2024-01-16T17:12:09.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.161+0000 7f1d2194c700 1 --2- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1d0c02ffb0 0x7f1d0c032470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:09.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.161+0000 7f1d2194c700 1 --2- 172.21.15.190:0/2238997135 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d1c0f6240 0x7f1d1c1064d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:09.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.161+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 >> 172.21.15.190:0/2238997135 conn(0x7f1d1c0f19c0 msgr2=0x7f1d1c0f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:09.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.162+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 shutdown_connections 2024-01-16T17:12:09.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:09.162+0000 7f1d2194c700 1 -- 172.21.15.190:0/2238997135 wait complete. 2024-01-16T17:12:09.144 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:09.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:07.793886+0000 mgr.smithi154.sllivz (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:09.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:09.161706+0000 mon.smithi154 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.190:0/2238997135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:10.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:10.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:11.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 -- 172.21.15.190:0/2082015898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 msgr2=0x7f93040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:11.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 --2- 172.21.15.190:0/2082015898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 0x7f93040f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9300009fb0 tx=0x7f9300009b30).stop 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 -- 172.21.15.190:0/2082015898 shutdown_connections 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 --2- 172.21.15.190:0/2082015898 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 0x7f93040f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 -- 172.21.15.190:0/2082015898 >> 172.21.15.190:0/2082015898 conn(0x7f93040f09a0 msgr2=0x7f93040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 -- 172.21.15.190:0/2082015898 shutdown_connections 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 -- 172.21.15.190:0/2082015898 wait complete. 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 Processor -- start 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.659+0000 7f930a24c700 1 -- start start 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f930a24c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 0x7f93041091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f930a24c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9304060220 con 0x7f93040f51a0 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f930924a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 0x7f93041091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f930924a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 0x7f93041091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52914/0 (socket says 172.21.15.190:52914) 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f930924a700 1 -- 172.21.15.190:0/240116369 learned_addr learned my addr 172.21.15.190:0/240116369 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f930924a700 1 -- 172.21.15.190:0/240116369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9300005040 con 0x7f93040f51a0 2024-01-16T17:12:11.643 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f930924a700 1 --2- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 0x7f93041091d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9300013510 tx=0x7f930000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:11.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f92f9ffb700 1 -- 172.21.15.190:0/240116369 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9300009320 con 0x7f93040f51a0 2024-01-16T17:12:11.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9304060420 con 0x7f93040f51a0 2024-01-16T17:12:11.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f92f9ffb700 1 -- 172.21.15.190:0/240116369 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9300013f00 con 0x7f93040f51a0 2024-01-16T17:12:11.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.660+0000 7f92f9ffb700 1 -- 172.21.15.190:0/240116369 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9300014dc0 con 0x7f93040f51a0 2024-01-16T17:12:11.644 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.661+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9304109880 con 0x7f93040f51a0 2024-01-16T17:12:11.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.661+0000 7f92f9ffb700 1 -- 172.21.15.190:0/240116369 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93000095f0 con 0x7f93040f51a0 2024-01-16T17:12:11.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.661+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9304059100 con 0x7f93040f51a0 2024-01-16T17:12:11.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.662+0000 7f92f9ffb700 1 --2- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f92f002ff60 0x7f92f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:11.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.662+0000 7f92f9ffb700 1 -- 172.21.15.190:0/240116369 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9300041040 con 0x7f93040f51a0 2024-01-16T17:12:11.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.662+0000 7f9308a49700 1 --2- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f92f002ff60 0x7f92f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:11.645 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.662+0000 7f9308a49700 1 --2- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f92f002ff60 0x7f92f0032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f92f40096a0 tx=0x7f92f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:11.648 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.666+0000 7f92f9ffb700 1 -- 172.21.15.190:0/240116369 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9300014420 con 0x7f93040f51a0 2024-01-16T17:12:11.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:09.794184+0000 mgr.smithi154.sllivz (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:11.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.872+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9304045ad0 con 0x7f93040f51a0 2024-01-16T17:12:11.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.873+0000 7f92f9ffb700 1 -- 172.21.15.190:0/240116369 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9300010230 con 0x7f93040f51a0 2024-01-16T17:12:11.855 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:11.855 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:11.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.874+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f92f002ff60 msgr2=0x7f92f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:11.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.874+0000 7f930a24c700 1 --2- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f92f002ff60 0x7f92f0032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f92f40096a0 tx=0x7f92f4006b40).stop 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.874+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 msgr2=0x7f93041091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.874+0000 7f930a24c700 1 --2- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 0x7f93041091d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9300013510 tx=0x7f930000b040).stop 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.874+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 shutdown_connections 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.875+0000 7f930a24c700 1 --2- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f92f002ff60 0x7f92f0032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.875+0000 7f930a24c700 1 --2- 172.21.15.190:0/240116369 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93040f51a0 0x7f93041091d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.875+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 >> 172.21.15.190:0/240116369 conn(0x7f93040f09a0 msgr2=0x7f93040f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.875+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 shutdown_connections 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:11.875+0000 7f930a24c700 1 -- 172.21.15.190:0/240116369 wait complete. 2024-01-16T17:12:11.857 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:12.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:12 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:11.874882+0000 mon.smithi154 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.190:0/240116369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:13.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:13.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:13.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:11.794599+0000 mgr.smithi154.sllivz (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:14.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.423+0000 7fc3549c5700 1 -- 172.21.15.190:0/826840812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 msgr2=0x7fc3500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:14.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.423+0000 7fc3549c5700 1 --2- 172.21.15.190:0/826840812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 0x7fc3500f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc344009fb0 tx=0x7fc344009b30).stop 2024-01-16T17:12:14.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.424+0000 7fc3549c5700 1 -- 172.21.15.190:0/826840812 shutdown_connections 2024-01-16T17:12:14.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.424+0000 7fc3549c5700 1 --2- 172.21.15.190:0/826840812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 0x7fc3500f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:14.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.424+0000 7fc3549c5700 1 -- 172.21.15.190:0/826840812 >> 172.21.15.190:0/826840812 conn(0x7fc3500f0b70 msgr2=0x7fc3500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:14.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.424+0000 7fc3549c5700 1 -- 172.21.15.190:0/826840812 shutdown_connections 2024-01-16T17:12:14.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.424+0000 7fc3549c5700 1 -- 172.21.15.190:0/826840812 wait complete. 2024-01-16T17:12:14.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.424+0000 7fc3549c5700 1 Processor -- start 2024-01-16T17:12:14.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.424+0000 7fc3549c5700 1 -- start start 2024-01-16T17:12:14.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc3549c5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 0x7fc350109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:14.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc3549c5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc350060220 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc34e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 0x7fc350109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc34e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 0x7fc350109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35968/0 (socket says 172.21.15.190:35968) 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc34e59c700 1 -- 172.21.15.190:0/846043370 learned_addr learned my addr 172.21.15.190:0/846043370 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc34e59c700 1 -- 172.21.15.190:0/846043370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc344005040 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc34e59c700 1 --2- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 0x7fc350109450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc344014000 tx=0x7fc34400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc33effd700 1 -- 172.21.15.190:0/846043370 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3440095f0 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc350060420 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc33effd700 1 -- 172.21.15.190:0/846043370 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc344007e70 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.425+0000 7fc33effd700 1 -- 172.21.15.190:0/846043370 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc344014ea0 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.426+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc350109b00 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.426+0000 7fc33effd700 1 -- 172.21.15.190:0/846043370 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc344025070 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.426+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc350059100 con 0x7fc3500f5410 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.426+0000 7fc33effd700 1 --2- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc33802ff60 0x7fc338032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:14.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.426+0000 7fc33effd700 1 -- 172.21.15.190:0/846043370 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc344040a30 con 0x7fc3500f5410 2024-01-16T17:12:14.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.427+0000 7fc34dd9b700 1 --2- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc33802ff60 0x7fc338032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:14.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.427+0000 7fc34dd9b700 1 --2- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc33802ff60 0x7fc338032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc3400096a0 tx=0x7fc340006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:14.413 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.432+0000 7fc33effd700 1 -- 172.21.15.190:0/846043370 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3440079c0 con 0x7fc3500f5410 2024-01-16T17:12:14.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.639+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc350045ad0 con 0x7fc3500f5410 2024-01-16T17:12:14.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.640+0000 7fc33effd700 1 -- 172.21.15.190:0/846043370 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc34400f300 con 0x7fc3500f5410 2024-01-16T17:12:14.622 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:14.622 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc33802ff60 msgr2=0x7fc338032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 --2- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc33802ff60 0x7fc338032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc3400096a0 tx=0x7fc340006b40).stop 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 msgr2=0x7fc350109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 --2- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 0x7fc350109450 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc344014000 tx=0x7fc34400b040).stop 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 shutdown_connections 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 --2- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc33802ff60 0x7fc338032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 --2- 172.21.15.190:0/846043370 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3500f5410 0x7fc350109450 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 >> 172.21.15.190:0/846043370 conn(0x7fc3500f0b70 msgr2=0x7fc3500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 shutdown_connections 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:14.641+0000 7fc3549c5700 1 -- 172.21.15.190:0/846043370 wait complete. 2024-01-16T17:12:14.624 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:15.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:13.794904+0000 mgr.smithi154.sllivz (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:15.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:14.641771+0000 mon.smithi154 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.190:0/846043370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:16.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:16.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:17.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 -- 172.21.15.190:0/3796184396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 msgr2=0x7f6be00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:17.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 --2- 172.21.15.190:0/3796184396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 0x7f6be00f90f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f6bd0009fb0 tx=0x7f6bd0009b30).stop 2024-01-16T17:12:17.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 -- 172.21.15.190:0/3796184396 shutdown_connections 2024-01-16T17:12:17.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 --2- 172.21.15.190:0/3796184396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 0x7f6be00f90f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:17.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 -- 172.21.15.190:0/3796184396 >> 172.21.15.190:0/3796184396 conn(0x7f6be00f0b70 msgr2=0x7f6be00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:17.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 -- 172.21.15.190:0/3796184396 shutdown_connections 2024-01-16T17:12:17.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 -- 172.21.15.190:0/3796184396 wait complete. 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 Processor -- start 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.164+0000 7f6be860a700 1 -- start start 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be860a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 0x7f6be01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be860a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0106bd0 con 0x7f6be0060280 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be63a6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 0x7f6be01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be63a6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 0x7f6be01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35982/0 (socket says 172.21.15.190:35982) 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be63a6700 1 -- 172.21.15.190:0/1720268312 learned_addr learned my addr 172.21.15.190:0/1720268312 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be63a6700 1 -- 172.21.15.190:0/1720268312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd0005040 con 0x7f6be0060280 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be63a6700 1 --2- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 0x7f6be01066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f6bd0013510 tx=0x7f6bd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6bd6ffd700 1 -- 172.21.15.190:0/1720268312 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd0009320 con 0x7f6be0060280 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be0106dd0 con 0x7f6be0060280 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6bd6ffd700 1 -- 172.21.15.190:0/1720268312 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bd0013f00 con 0x7f6be0060280 2024-01-16T17:12:17.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6bd6ffd700 1 -- 172.21.15.190:0/1720268312 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bd0014dc0 con 0x7f6be0060280 2024-01-16T17:12:17.148 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.165+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be0107260 con 0x7f6be0060280 2024-01-16T17:12:17.148 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.166+0000 7f6bd6ffd700 1 -- 172.21.15.190:0/1720268312 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bd00095f0 con 0x7f6be0060280 2024-01-16T17:12:17.149 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.166+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be0045ad0 con 0x7f6be0060280 2024-01-16T17:12:17.149 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.166+0000 7f6bd6ffd700 1 --2- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bcc02ff60 0x7f6bcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:17.149 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.167+0000 7f6bd6ffd700 1 -- 172.21.15.190:0/1720268312 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bd00410c0 con 0x7f6be0060280 2024-01-16T17:12:17.149 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.167+0000 7f6be5ba5700 1 --2- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bcc02ff60 0x7f6bcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:17.149 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.167+0000 7f6be5ba5700 1 --2- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bcc02ff60 0x7f6bcc032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f6bdc0096a0 tx=0x7f6bdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:17.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.171+0000 7f6bd6ffd700 1 -- 172.21.15.190:0/1720268312 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bd001c790 con 0x7f6be0060280 2024-01-16T17:12:17.360 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.379+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6be0108110 con 0x7f6be0060280 2024-01-16T17:12:17.361 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.380+0000 7f6bd6ffd700 1 -- 172.21.15.190:0/1720268312 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6bd0010230 con 0x7f6be0060280 2024-01-16T17:12:17.361 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:17.361 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bcc02ff60 msgr2=0x7f6bcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 --2- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bcc02ff60 0x7f6bcc032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f6bdc0096a0 tx=0x7f6bdc006b40).stop 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 msgr2=0x7f6be01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 --2- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 0x7f6be01066e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f6bd0013510 tx=0x7f6bd000b040).stop 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 shutdown_connections 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 --2- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6bcc02ff60 0x7f6bcc032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 --2- 172.21.15.190:0/1720268312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6be0060280 0x7f6be01066e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 >> 172.21.15.190:0/1720268312 conn(0x7f6be00f0b70 msgr2=0x7f6be00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 shutdown_connections 2024-01-16T17:12:17.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:17.382+0000 7f6be860a700 1 -- 172.21.15.190:0/1720268312 wait complete. 2024-01-16T17:12:17.364 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:17.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:15.795265+0000 mgr.smithi154.sllivz (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:17.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:17.380901+0000 mon.smithi154 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.190:0/1720268312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:18.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:18.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:19.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:17.795623+0000 mgr.smithi154.sllivz (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:19.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 -- 172.21.15.190:0/1793852165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 msgr2=0x7f94d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:19.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 --2- 172.21.15.190:0/1793852165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 0x7f94d80f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f94cc009fb0 tx=0x7f94cc009b30).stop 2024-01-16T17:12:19.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 -- 172.21.15.190:0/1793852165 shutdown_connections 2024-01-16T17:12:19.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 --2- 172.21.15.190:0/1793852165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 0x7f94d80f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:19.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 -- 172.21.15.190:0/1793852165 >> 172.21.15.190:0/1793852165 conn(0x7f94d80f0b70 msgr2=0x7f94d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:19.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 -- 172.21.15.190:0/1793852165 shutdown_connections 2024-01-16T17:12:19.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 -- 172.21.15.190:0/1793852165 wait complete. 2024-01-16T17:12:19.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 Processor -- start 2024-01-16T17:12:19.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.939+0000 7f94deb6c700 1 -- start start 2024-01-16T17:12:19.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94deb6c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 0x7f94d8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:19.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94deb6c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d8105b20 con 0x7f94d80f5410 2024-01-16T17:12:19.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94dc908700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 0x7f94d8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:19.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94dc908700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 0x7f94d8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36000/0 (socket says 172.21.15.190:36000) 2024-01-16T17:12:19.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94dc908700 1 -- 172.21.15.190:0/3374665279 learned_addr learned my addr 172.21.15.190:0/3374665279 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94dc908700 1 -- 172.21.15.190:0/3374665279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc005040 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94dc908700 1 --2- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 0x7f94d8105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f94cc013510 tx=0x7f94cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94d57fa700 1 -- 172.21.15.190:0/3374665279 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94cc009320 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94d57fa700 1 -- 172.21.15.190:0/3374665279 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94cc013f00 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94d57fa700 1 -- 172.21.15.190:0/3374665279 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94cc014d90 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.940+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d8105d20 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.941+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d81061b0 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.941+0000 7f94d57fa700 1 -- 172.21.15.190:0/3374665279 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94cc0095f0 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.941+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d8045ad0 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.941+0000 7f94d57fa700 1 --2- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f94c802ff60 0x7f94c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.941+0000 7f94d57fa700 1 -- 172.21.15.190:0/3374665279 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94cc041000 con 0x7f94d80f5410 2024-01-16T17:12:19.923 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.942+0000 7f94d7fff700 1 --2- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f94c802ff60 0x7f94c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:19.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.942+0000 7f94d7fff700 1 --2- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f94c802ff60 0x7f94c8032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f94c40096a0 tx=0x7f94c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:19.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:19.946+0000 7f94d57fa700 1 -- 172.21.15.190:0/3374665279 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94cc0268c0 con 0x7f94d80f5410 2024-01-16T17:12:20.133 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.153+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94d80fae60 con 0x7f94d80f5410 2024-01-16T17:12:20.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.153+0000 7f94d57fa700 1 -- 172.21.15.190:0/3374665279 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f94cc010230 con 0x7f94d80f5410 2024-01-16T17:12:20.134 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:20.134 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f94c802ff60 msgr2=0x7f94c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 --2- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f94c802ff60 0x7f94c8032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f94c40096a0 tx=0x7f94c4006b40).stop 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 msgr2=0x7f94d8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 --2- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 0x7f94d8105630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f94cc013510 tx=0x7f94cc00b040).stop 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 shutdown_connections 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 --2- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f94c802ff60 0x7f94c8032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 --2- 172.21.15.190:0/3374665279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94d80f5410 0x7f94d8105630 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 >> 172.21.15.190:0/3374665279 conn(0x7f94d80f0b70 msgr2=0x7f94d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 shutdown_connections 2024-01-16T17:12:20.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:20.155+0000 7f94deb6c700 1 -- 172.21.15.190:0/3374665279 wait complete. 2024-01-16T17:12:20.137 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:20.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:20 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:20.154405+0000 mon.smithi154 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.190:0/3374665279' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:21.523 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:21.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:21.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:19.795951+0000 mgr.smithi154.sllivz (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 -- 172.21.15.190:0/3142915352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 msgr2=0x7f57340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 --2- 172.21.15.190:0/3142915352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 0x7f57340f55c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f5724009fb0 tx=0x7f5724009b30).stop 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 -- 172.21.15.190:0/3142915352 shutdown_connections 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 --2- 172.21.15.190:0/3142915352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 0x7f57340f55c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 -- 172.21.15.190:0/3142915352 >> 172.21.15.190:0/3142915352 conn(0x7f57340f09a0 msgr2=0x7f57340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 -- 172.21.15.190:0/3142915352 shutdown_connections 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 -- 172.21.15.190:0/3142915352 wait complete. 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 Processor -- start 2024-01-16T17:12:22.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.710+0000 7f573359e700 1 -- start start 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573359e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 0x7f57341091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573359e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734060240 con 0x7f57340f51a0 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 0x7f57341091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 0x7f57341091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44972/0 (socket says 172.21.15.190:44972) 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573259c700 1 -- 172.21.15.190:0/1384903155 learned_addr learned my addr 172.21.15.190:0/1384903155 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573259c700 1 -- 172.21.15.190:0/1384903155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5724005040 con 0x7f57340f51a0 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573259c700 1 --2- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 0x7f57341091b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5724013510 tx=0x7f572400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f5722ffd700 1 -- 172.21.15.190:0/1384903155 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5724009320 con 0x7f57340f51a0 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5734060440 con 0x7f57340f51a0 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f5722ffd700 1 -- 172.21.15.190:0/1384903155 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5724013f00 con 0x7f57340f51a0 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f5722ffd700 1 -- 172.21.15.190:0/1384903155 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5724014dc0 con 0x7f57340f51a0 2024-01-16T17:12:22.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.711+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5734109860 con 0x7f57340f51a0 2024-01-16T17:12:22.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.712+0000 7f5722ffd700 1 -- 172.21.15.190:0/1384903155 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5724014420 con 0x7f57340f51a0 2024-01-16T17:12:22.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.712+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5734045ad0 con 0x7f57340f51a0 2024-01-16T17:12:22.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.712+0000 7f5722ffd700 1 --2- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f571c02ffb0 0x7f571c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:22.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.712+0000 7f5722ffd700 1 -- 172.21.15.190:0/1384903155 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5724041110 con 0x7f57340f51a0 2024-01-16T17:12:22.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.712+0000 7f5731d9b700 1 --2- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f571c02ffb0 0x7f571c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:22.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.713+0000 7f5731d9b700 1 --2- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f571c02ffb0 0x7f571c032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f57280096a0 tx=0x7f5728006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:22.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.717+0000 7f5722ffd700 1 -- 172.21.15.190:0/1384903155 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5724012330 con 0x7f57340f51a0 2024-01-16T17:12:22.902 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.922+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f573410a080 con 0x7f57340f51a0 2024-01-16T17:12:22.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.923+0000 7f5722ffd700 1 -- 172.21.15.190:0/1384903155 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f572401cb10 con 0x7f57340f51a0 2024-01-16T17:12:22.903 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:22.903 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f571c02ffb0 msgr2=0x7f571c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 --2- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f571c02ffb0 0x7f571c032470 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f57280096a0 tx=0x7f5728006b40).stop 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 msgr2=0x7f57341091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 --2- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 0x7f57341091b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f5724013510 tx=0x7f572400b040).stop 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 shutdown_connections 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 --2- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f571c02ffb0 0x7f571c032470 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 --2- 172.21.15.190:0/1384903155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57340f51a0 0x7f57341091b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 >> 172.21.15.190:0/1384903155 conn(0x7f57340f09a0 msgr2=0x7f57340f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:22.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 shutdown_connections 2024-01-16T17:12:22.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:22.924+0000 7f573359e700 1 -- 172.21.15.190:0/1384903155 wait complete. 2024-01-16T17:12:22.906 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:23.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:21.796206+0000 mgr.smithi154.sllivz (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:23.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:22.923550+0000 mon.smithi154 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.190:0/1384903155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:24.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:24.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:25.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:23.796476+0000 mgr.smithi154.sllivz (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:26.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.643+0000 7f714e8de700 1 -- 172.21.15.190:0/4245774752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 msgr2=0x7f71480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.643+0000 7f714e8de700 1 --2- 172.21.15.190:0/4245774752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 0x7f71480f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f7138009fb0 tx=0x7f7138009b30).stop 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.643+0000 7f714e8de700 1 -- 172.21.15.190:0/4245774752 shutdown_connections 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.643+0000 7f714e8de700 1 --2- 172.21.15.190:0/4245774752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 0x7f71480f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.643+0000 7f714e8de700 1 -- 172.21.15.190:0/4245774752 >> 172.21.15.190:0/4245774752 conn(0x7f71480f0b70 msgr2=0x7f71480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.643+0000 7f714e8de700 1 -- 172.21.15.190:0/4245774752 shutdown_connections 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.643+0000 7f714e8de700 1 -- 172.21.15.190:0/4245774752 wait complete. 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.644+0000 7f714e8de700 1 Processor -- start 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.644+0000 7f714e8de700 1 -- start start 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.644+0000 7f714e8de700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 0x7f7148108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:26.625 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.644+0000 7f714e8de700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7148108e00 con 0x7f71480f5410 2024-01-16T17:12:26.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f7147fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 0x7f7148108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:26.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f7147fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 0x7f7148108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:45000/0 (socket says 172.21.15.190:45000) 2024-01-16T17:12:26.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f7147fff700 1 -- 172.21.15.190:0/923581206 learned_addr learned my addr 172.21.15.190:0/923581206 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:26.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f7147fff700 1 -- 172.21.15.190:0/923581206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7138005040 con 0x7f71480f5410 2024-01-16T17:12:26.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f7147fff700 1 --2- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 0x7f7148108910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7138013510 tx=0x7f713800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:26.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f7144ff9700 1 -- 172.21.15.190:0/923581206 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7138009320 con 0x7f71480f5410 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7148109000 con 0x7f71480f5410 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f7144ff9700 1 -- 172.21.15.190:0/923581206 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7138013f00 con 0x7f71480f5410 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f7144ff9700 1 -- 172.21.15.190:0/923581206 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7138014d90 con 0x7f71480f5410 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.645+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7148109490 con 0x7f71480f5410 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.646+0000 7f7144ff9700 1 -- 172.21.15.190:0/923581206 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7138014420 con 0x7f71480f5410 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.646+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7148045ad0 con 0x7f71480f5410 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.646+0000 7f7144ff9700 1 --2- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f713402ff60 0x7f7134032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.646+0000 7f7144ff9700 1 -- 172.21.15.190:0/923581206 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7138041050 con 0x7f71480f5410 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.646+0000 7f71477fe700 1 --2- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f713402ff60 0x7f7134032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:26.628 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.647+0000 7f71477fe700 1 --2- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f713402ff60 0x7f7134032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f71300096a0 tx=0x7f7130006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:26.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.651+0000 7f7144ff9700 1 -- 172.21.15.190:0/923581206 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71380095f0 con 0x7f71480f5410 2024-01-16T17:12:26.836 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.856+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71480fae60 con 0x7f71480f5410 2024-01-16T17:12:26.837 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.856+0000 7f7144ff9700 1 -- 172.21.15.190:0/923581206 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f713801db70 con 0x7f71480f5410 2024-01-16T17:12:26.837 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:26.837 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:26.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f713402ff60 msgr2=0x7f7134032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:26.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 --2- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f713402ff60 0x7f7134032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f71300096a0 tx=0x7f7130006b40).stop 2024-01-16T17:12:26.838 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 msgr2=0x7f7148108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 --2- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 0x7f7148108910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7138013510 tx=0x7f713800b040).stop 2024-01-16T17:12:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 shutdown_connections 2024-01-16T17:12:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 --2- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f713402ff60 0x7f7134032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 --2- 172.21.15.190:0/923581206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71480f5410 0x7f7148108910 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 >> 172.21.15.190:0/923581206 conn(0x7f71480f0b70 msgr2=0x7f714805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 shutdown_connections 2024-01-16T17:12:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:26.858+0000 7f714e8de700 1 -- 172.21.15.190:0/923581206 wait complete. 2024-01-16T17:12:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:27.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:25.796740+0000 mgr.smithi154.sllivz (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:27.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:26.857228+0000 mon.smithi154 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.190:0/923581206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:28.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:28.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:29.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 -- 172.21.15.190:0/3314729036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 msgr2=0x7f11800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:29.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 --2- 172.21.15.190:0/3314729036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 0x7f11800f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1170009fb0 tx=0x7f1170009b30).stop 2024-01-16T17:12:29.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 -- 172.21.15.190:0/3314729036 shutdown_connections 2024-01-16T17:12:29.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 --2- 172.21.15.190:0/3314729036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 0x7f11800f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 -- 172.21.15.190:0/3314729036 >> 172.21.15.190:0/3314729036 conn(0x7f11800f0b70 msgr2=0x7f11800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 -- 172.21.15.190:0/3314729036 shutdown_connections 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 -- 172.21.15.190:0/3314729036 wait complete. 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 Processor -- start 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.399+0000 7f1188153700 1 -- start start 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.400+0000 7f1188153700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 0x7f11800fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.400+0000 7f1188153700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11800fb820 con 0x7f11800f5410 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.400+0000 7f1185eef700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 0x7f11800fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.400+0000 7f1185eef700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 0x7f11800fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:45014/0 (socket says 172.21.15.190:45014) 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.400+0000 7f1185eef700 1 -- 172.21.15.190:0/2081332418 learned_addr learned my addr 172.21.15.190:0/2081332418 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.400+0000 7f1185eef700 1 -- 172.21.15.190:0/2081332418 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1170005040 con 0x7f11800f5410 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.400+0000 7f1185eef700 1 --2- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 0x7f11800fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1170013510 tx=0x7f117000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.400+0000 7f11767fc700 1 -- 172.21.15.190:0/2081332418 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1170009320 con 0x7f11800f5410 2024-01-16T17:12:29.382 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.401+0000 7f11767fc700 1 -- 172.21.15.190:0/2081332418 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1170013f00 con 0x7f11800f5410 2024-01-16T17:12:29.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.401+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11800fb9c0 con 0x7f11800f5410 2024-01-16T17:12:29.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.401+0000 7f11767fc700 1 -- 172.21.15.190:0/2081332418 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1170014d90 con 0x7f11800f5410 2024-01-16T17:12:29.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.401+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11800fbe50 con 0x7f11800f5410 2024-01-16T17:12:29.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.401+0000 7f11767fc700 1 -- 172.21.15.190:0/2081332418 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11700095f0 con 0x7f11800f5410 2024-01-16T17:12:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.401+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1180059100 con 0x7f11800f5410 2024-01-16T17:12:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.401+0000 7f11767fc700 1 --2- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f116c02ff60 0x7f116c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.402+0000 7f11767fc700 1 -- 172.21.15.190:0/2081332418 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1170041080 con 0x7f11800f5410 2024-01-16T17:12:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.402+0000 7f11856ee700 1 --2- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f116c02ff60 0x7f116c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.402+0000 7f11856ee700 1 --2- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f116c02ff60 0x7f116c032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f117c0096a0 tx=0x7f117c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.406+0000 7f11767fc700 1 -- 172.21.15.190:0/2081332418 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1170014420 con 0x7f11800f5410 2024-01-16T17:12:29.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.617+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11800fae60 con 0x7f11800f5410 2024-01-16T17:12:29.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.618+0000 7f11767fc700 1 -- 172.21.15.190:0/2081332418 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f117001d7f0 con 0x7f11800f5410 2024-01-16T17:12:29.598 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:29.599 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:29.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f116c02ff60 msgr2=0x7f116c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:29.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 --2- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f116c02ff60 0x7f116c032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f117c0096a0 tx=0x7f117c006b40).stop 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 msgr2=0x7f11800fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 --2- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 0x7f11800fd3a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1170013510 tx=0x7f117000b040).stop 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 shutdown_connections 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 --2- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f116c02ff60 0x7f116c032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 --2- 172.21.15.190:0/2081332418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11800f5410 0x7f11800fd3a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 >> 172.21.15.190:0/2081332418 conn(0x7f11800f0b70 msgr2=0x7f118005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 shutdown_connections 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:29.619+0000 7f1188153700 1 -- 172.21.15.190:0/2081332418 wait complete. 2024-01-16T17:12:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:29.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:27.796993+0000 mgr.smithi154.sllivz (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:30.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:30 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:29.618740+0000 mon.smithi154 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.190:0/2081332418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:30.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:30.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:31.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:29.797326+0000 mgr.smithi154.sllivz (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:32.255 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.274+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/832873752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 msgr2=0x7f6f340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.274+0000 7f6f3c4f9700 1 --2- 172.21.15.190:0/832873752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 0x7f6f340f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f6f24009fb0 tx=0x7f6f24009b30).stop 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.274+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/832873752 shutdown_connections 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.274+0000 7f6f3c4f9700 1 --2- 172.21.15.190:0/832873752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 0x7f6f340f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.274+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/832873752 >> 172.21.15.190:0/832873752 conn(0x7f6f340f0b70 msgr2=0x7f6f340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.274+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/832873752 shutdown_connections 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.274+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/832873752 wait complete. 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.275+0000 7f6f3c4f9700 1 Processor -- start 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.275+0000 7f6f3c4f9700 1 -- start start 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.275+0000 7f6f3c4f9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 0x7f6f34106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.275+0000 7f6f3c4f9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f34106c10 con 0x7f6f340f5410 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.275+0000 7f6f3a295700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 0x7f6f34106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:32.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.275+0000 7f6f3a295700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 0x7f6f34106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:45030/0 (socket says 172.21.15.190:45030) 2024-01-16T17:12:32.257 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.275+0000 7f6f3a295700 1 -- 172.21.15.190:0/2468042168 learned_addr learned my addr 172.21.15.190:0/2468042168 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.276+0000 7f6f3a295700 1 -- 172.21.15.190:0/2468042168 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f24005040 con 0x7f6f340f5410 2024-01-16T17:12:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.276+0000 7f6f3a295700 1 --2- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 0x7f6f34106720 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f6f24013510 tx=0x7f6f2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.276+0000 7f6f2affd700 1 -- 172.21.15.190:0/2468042168 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f24009320 con 0x7f6f340f5410 2024-01-16T17:12:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.276+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f34106e10 con 0x7f6f340f5410 2024-01-16T17:12:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.276+0000 7f6f2affd700 1 -- 172.21.15.190:0/2468042168 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f24013f00 con 0x7f6f340f5410 2024-01-16T17:12:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.276+0000 7f6f2affd700 1 -- 172.21.15.190:0/2468042168 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f24014d90 con 0x7f6f340f5410 2024-01-16T17:12:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.276+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f341072a0 con 0x7f6f340f5410 2024-01-16T17:12:32.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.277+0000 7f6f2affd700 1 -- 172.21.15.190:0/2468042168 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f240095f0 con 0x7f6f340f5410 2024-01-16T17:12:32.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.277+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f34045ad0 con 0x7f6f340f5410 2024-01-16T17:12:32.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.277+0000 7f6f2affd700 1 --2- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f2002ff60 0x7f6f20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:32.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.277+0000 7f6f2affd700 1 -- 172.21.15.190:0/2468042168 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f24041080 con 0x7f6f340f5410 2024-01-16T17:12:32.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.277+0000 7f6f39a94700 1 --2- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f2002ff60 0x7f6f20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:32.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.277+0000 7f6f39a94700 1 --2- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f2002ff60 0x7f6f20032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f6f300096a0 tx=0x7f6f30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:32.262 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.281+0000 7f6f2affd700 1 -- 172.21.15.190:0/2468042168 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f24014420 con 0x7f6f340f5410 2024-01-16T17:12:32.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.487+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f34108150 con 0x7f6f340f5410 2024-01-16T17:12:32.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.488+0000 7f6f2affd700 1 -- 172.21.15.190:0/2468042168 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6f24010230 con 0x7f6f340f5410 2024-01-16T17:12:32.468 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:32.469 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:32.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f2002ff60 msgr2=0x7f6f20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:32.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 --2- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f2002ff60 0x7f6f20032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f6f300096a0 tx=0x7f6f30006b40).stop 2024-01-16T17:12:32.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 msgr2=0x7f6f34106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:32.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 --2- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 0x7f6f34106720 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f6f24013510 tx=0x7f6f2400b040).stop 2024-01-16T17:12:32.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 shutdown_connections 2024-01-16T17:12:32.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 --2- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6f2002ff60 0x7f6f20032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:32.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 --2- 172.21.15.190:0/2468042168 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f340f5410 0x7f6f34106720 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:32.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 >> 172.21.15.190:0/2468042168 conn(0x7f6f340f0b70 msgr2=0x7f6f3405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:32.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 shutdown_connections 2024-01-16T17:12:32.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:32.489+0000 7f6f3c4f9700 1 -- 172.21.15.190:0/2468042168 wait complete. 2024-01-16T17:12:32.471 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:33.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:33.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:33.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:31.797696+0000 mgr.smithi154.sllivz (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:33.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:32.488554+0000 mon.smithi154 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.190:0/2468042168' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:34.999+0000 7f55652e8700 1 -- 172.21.15.190:0/3077445721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 msgr2=0x7f55600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:34.999+0000 7f55652e8700 1 --2- 172.21.15.190:0/3077445721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 0x7f55600f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f5548009fb0 tx=0x7f5548009b30).stop 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:34.999+0000 7f55652e8700 1 -- 172.21.15.190:0/3077445721 shutdown_connections 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:34.999+0000 7f55652e8700 1 --2- 172.21.15.190:0/3077445721 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 0x7f55600f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:34.999+0000 7f55652e8700 1 -- 172.21.15.190:0/3077445721 >> 172.21.15.190:0/3077445721 conn(0x7f55600f0b70 msgr2=0x7f55600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:34.999+0000 7f55652e8700 1 -- 172.21.15.190:0/3077445721 shutdown_connections 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:34.999+0000 7f55652e8700 1 -- 172.21.15.190:0/3077445721 wait complete. 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f55652e8700 1 Processor -- start 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f55652e8700 1 -- start start 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f55652e8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 0x7f55601066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:34.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f55652e8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5560106bc0 con 0x7f55600f5410 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f555effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 0x7f55601066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f555effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 0x7f55601066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35122/0 (socket says 172.21.15.190:35122) 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f555effd700 1 -- 172.21.15.190:0/2619611157 learned_addr learned my addr 172.21.15.190:0/2619611157 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f555effd700 1 -- 172.21.15.190:0/2619611157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5548005040 con 0x7f55600f5410 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.000+0000 7f555effd700 1 --2- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 0x7f55601066d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5548013510 tx=0x7f554800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.001+0000 7f55577fe700 1 -- 172.21.15.190:0/2619611157 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5548009320 con 0x7f55600f5410 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.001+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5560106dc0 con 0x7f55600f5410 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.001+0000 7f55577fe700 1 -- 172.21.15.190:0/2619611157 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5548013f00 con 0x7f55600f5410 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.001+0000 7f55577fe700 1 -- 172.21.15.190:0/2619611157 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5548014d90 con 0x7f55600f5410 2024-01-16T17:12:34.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.001+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5560107250 con 0x7f55600f5410 2024-01-16T17:12:34.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.001+0000 7f55577fe700 1 -- 172.21.15.190:0/2619611157 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55480095f0 con 0x7f55600f5410 2024-01-16T17:12:34.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.001+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5560045ad0 con 0x7f55600f5410 2024-01-16T17:12:34.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.002+0000 7f55577fe700 1 --2- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f554c02ff60 0x7f554c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:34.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.002+0000 7f55577fe700 1 -- 172.21.15.190:0/2619611157 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f55480410a0 con 0x7f55600f5410 2024-01-16T17:12:34.984 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.002+0000 7f555e7fc700 1 --2- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f554c02ff60 0x7f554c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:34.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.002+0000 7f555e7fc700 1 --2- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f554c02ff60 0x7f554c032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f55500096a0 tx=0x7f5550006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:34.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.006+0000 7f55577fe700 1 -- 172.21.15.190:0/2619611157 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f554801d7f0 con 0x7f55600f5410 2024-01-16T17:12:35.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.213+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5560108100 con 0x7f55600f5410 2024-01-16T17:12:35.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.214+0000 7f55577fe700 1 -- 172.21.15.190:0/2619611157 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5548010230 con 0x7f55600f5410 2024-01-16T17:12:35.194 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:35.194 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f554c02ff60 msgr2=0x7f554c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 --2- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f554c02ff60 0x7f554c032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f55500096a0 tx=0x7f5550006b40).stop 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 msgr2=0x7f55601066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 --2- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 0x7f55601066d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f5548013510 tx=0x7f554800b040).stop 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 shutdown_connections 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 --2- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f554c02ff60 0x7f554c032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 --2- 172.21.15.190:0/2619611157 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55600f5410 0x7f55601066d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 >> 172.21.15.190:0/2619611157 conn(0x7f55600f0b70 msgr2=0x7f556005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 shutdown_connections 2024-01-16T17:12:35.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:35.215+0000 7f55652e8700 1 -- 172.21.15.190:0/2619611157 wait complete. 2024-01-16T17:12:35.197 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:35.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:33.798010+0000 mgr.smithi154.sllivz (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:35.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:35.214572+0000 mon.smithi154 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.190:0/2619611157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:36.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:36.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:37.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 -- 172.21.15.190:0/181795246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 msgr2=0x7f0cb00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:37.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 --2- 172.21.15.190:0/181795246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 0x7f0cb00f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f0c98009fb0 tx=0x7f0c98009b30).stop 2024-01-16T17:12:37.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 -- 172.21.15.190:0/181795246 shutdown_connections 2024-01-16T17:12:37.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 --2- 172.21.15.190:0/181795246 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 0x7f0cb00f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:37.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 -- 172.21.15.190:0/181795246 >> 172.21.15.190:0/181795246 conn(0x7f0cb00f1c70 msgr2=0x7f0cb00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:37.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 -- 172.21.15.190:0/181795246 shutdown_connections 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 -- 172.21.15.190:0/181795246 wait complete. 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 Processor -- start 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 -- start start 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.731+0000 7f0cb48bc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 0x7f0cb0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0cb48bc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0106c20 con 0x7f0cb00f6510 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0cae59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 0x7f0cb0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0cae59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 0x7f0cb0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35146/0 (socket says 172.21.15.190:35146) 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0cae59c700 1 -- 172.21.15.190:0/269822937 learned_addr learned my addr 172.21.15.190:0/269822937 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0cae59c700 1 -- 172.21.15.190:0/269822937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c98005040 con 0x7f0cb00f6510 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0cae59c700 1 --2- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 0x7f0cb0106730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0c98013510 tx=0x7f0c9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0ca6ffd700 1 -- 172.21.15.190:0/269822937 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c98009320 con 0x7f0cb00f6510 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb0106e20 con 0x7f0cb00f6510 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0ca6ffd700 1 -- 172.21.15.190:0/269822937 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c98013f00 con 0x7f0cb00f6510 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0ca6ffd700 1 -- 172.21.15.190:0/269822937 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c98014d90 con 0x7f0cb00f6510 2024-01-16T17:12:37.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.732+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb01072b0 con 0x7f0cb00f6510 2024-01-16T17:12:37.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.733+0000 7f0ca6ffd700 1 -- 172.21.15.190:0/269822937 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c980095f0 con 0x7f0cb00f6510 2024-01-16T17:12:37.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.733+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb0045ad0 con 0x7f0cb00f6510 2024-01-16T17:12:37.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.733+0000 7f0ca6ffd700 1 --2- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c9c02ff60 0x7f0c9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:37.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.733+0000 7f0ca6ffd700 1 -- 172.21.15.190:0/269822937 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c98040fc0 con 0x7f0cb00f6510 2024-01-16T17:12:37.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.733+0000 7f0cadd9b700 1 --2- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c9c02ff60 0x7f0c9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:37.715 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.734+0000 7f0cadd9b700 1 --2- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c9c02ff60 0x7f0c9c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0ca00096a0 tx=0x7f0ca0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:37.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.738+0000 7f0ca6ffd700 1 -- 172.21.15.190:0/269822937 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c98014420 con 0x7f0cb00f6510 2024-01-16T17:12:37.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.943+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cb00fbf60 con 0x7f0cb00f6510 2024-01-16T17:12:37.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.944+0000 7f0ca6ffd700 1 -- 172.21.15.190:0/269822937 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0c98010230 con 0x7f0cb00f6510 2024-01-16T17:12:37.924 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:37.924 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c9c02ff60 msgr2=0x7f0c9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 --2- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c9c02ff60 0x7f0c9c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f0ca00096a0 tx=0x7f0ca0006b40).stop 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 msgr2=0x7f0cb0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 --2- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 0x7f0cb0106730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0c98013510 tx=0x7f0c9800b040).stop 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 shutdown_connections 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 --2- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c9c02ff60 0x7f0c9c032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 --2- 172.21.15.190:0/269822937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cb00f6510 0x7f0cb0106730 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 >> 172.21.15.190:0/269822937 conn(0x7f0cb00f1c70 msgr2=0x7f0cb00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 shutdown_connections 2024-01-16T17:12:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:37.945+0000 7f0cb48bc700 1 -- 172.21.15.190:0/269822937 wait complete. 2024-01-16T17:12:37.927 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:37.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:35.798302+0000 mgr.smithi154.sllivz (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:38.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:38 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:37.944642+0000 mon.smithi154 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.190:0/269822937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:39.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:39.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:39.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:37.798638+0000 mgr.smithi154.sllivz (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 -- 172.21.15.190:0/1069452430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 msgr2=0x7f9c580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 --2- 172.21.15.190:0/1069452430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 0x7f9c580f55c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f9c48009fb0 tx=0x7f9c48009b30).stop 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 -- 172.21.15.190:0/1069452430 shutdown_connections 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 --2- 172.21.15.190:0/1069452430 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 0x7f9c580f55c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 -- 172.21.15.190:0/1069452430 >> 172.21.15.190:0/1069452430 conn(0x7f9c580f09a0 msgr2=0x7f9c580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 -- 172.21.15.190:0/1069452430 shutdown_connections 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 -- 172.21.15.190:0/1069452430 wait complete. 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 Processor -- start 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 -- start start 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 0x7f9c581064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.507+0000 7f9c57fff700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c581069d0 con 0x7f9c580f51a0 2024-01-16T17:12:40.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c56ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 0x7f9c581064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c56ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 0x7f9c581064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35160/0 (socket says 172.21.15.190:35160) 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c56ffd700 1 -- 172.21.15.190:0/877846833 learned_addr learned my addr 172.21.15.190:0/877846833 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c56ffd700 1 -- 172.21.15.190:0/877846833 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c48005040 con 0x7f9c580f51a0 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c56ffd700 1 --2- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 0x7f9c581064e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f9c48013510 tx=0x7f9c4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c3f7fe700 1 -- 172.21.15.190:0/877846833 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c48009320 con 0x7f9c580f51a0 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c58106bd0 con 0x7f9c580f51a0 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c3f7fe700 1 -- 172.21.15.190:0/877846833 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c48013f00 con 0x7f9c580f51a0 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c3f7fe700 1 -- 172.21.15.190:0/877846833 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c48014dc0 con 0x7f9c580f51a0 2024-01-16T17:12:40.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.508+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c58107060 con 0x7f9c580f51a0 2024-01-16T17:12:40.491 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.509+0000 7f9c3f7fe700 1 -- 172.21.15.190:0/877846833 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c4801d3d0 con 0x7f9c580f51a0 2024-01-16T17:12:40.491 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.509+0000 7f9c3f7fe700 1 --2- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9c4002ff60 0x7f9c40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:40.491 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.509+0000 7f9c3f7fe700 1 -- 172.21.15.190:0/877846833 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c48041030 con 0x7f9c580f51a0 2024-01-16T17:12:40.491 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.509+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c58045ad0 con 0x7f9c580f51a0 2024-01-16T17:12:40.491 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.509+0000 7f9c567fc700 1 --2- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9c4002ff60 0x7f9c40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:40.491 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.510+0000 7f9c567fc700 1 --2- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9c4002ff60 0x7f9c40032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9c4c0096a0 tx=0x7f9c4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:40.494 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.514+0000 7f9c3f7fe700 1 -- 172.21.15.190:0/877846833 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c48012920 con 0x7f9c580f51a0 2024-01-16T17:12:40.702 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.721+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c580facd0 con 0x7f9c580f51a0 2024-01-16T17:12:40.702 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.722+0000 7f9c3f7fe700 1 -- 172.21.15.190:0/877846833 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9c4802b020 con 0x7f9c580f51a0 2024-01-16T17:12:40.703 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:40.703 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:40.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9c4002ff60 msgr2=0x7f9c40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:40.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 --2- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9c4002ff60 0x7f9c40032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9c4c0096a0 tx=0x7f9c4c006b40).stop 2024-01-16T17:12:40.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 msgr2=0x7f9c581064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:40.704 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 --2- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 0x7f9c581064e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f9c48013510 tx=0x7f9c4800b040).stop 2024-01-16T17:12:40.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 shutdown_connections 2024-01-16T17:12:40.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 --2- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9c4002ff60 0x7f9c40032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:40.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 --2- 172.21.15.190:0/877846833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9c580f51a0 0x7f9c581064e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:40.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 >> 172.21.15.190:0/877846833 conn(0x7f9c580f09a0 msgr2=0x7f9c5805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:40.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 shutdown_connections 2024-01-16T17:12:40.705 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:40.724+0000 7f9c57fff700 1 -- 172.21.15.190:0/877846833 wait complete. 2024-01-16T17:12:40.705 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:41.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:39.798983+0000 mgr.smithi154.sllivz (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:41.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:40.722735+0000 mon.smithi154 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.190:0/877846833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:42.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:42.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:43.304 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.321+0000 7f95f5cc4700 1 -- 172.21.15.190:0/1043350774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 msgr2=0x7f95f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:43.304 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.321+0000 7f95f5cc4700 1 --2- 172.21.15.190:0/1043350774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 0x7f95f00f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f95d8009fb0 tx=0x7f95d8009b30).stop 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.321+0000 7f95f5cc4700 1 -- 172.21.15.190:0/1043350774 shutdown_connections 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.321+0000 7f95f5cc4700 1 --2- 172.21.15.190:0/1043350774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 0x7f95f00f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.321+0000 7f95f5cc4700 1 -- 172.21.15.190:0/1043350774 >> 172.21.15.190:0/1043350774 conn(0x7f95f00f0b70 msgr2=0x7f95f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.321+0000 7f95f5cc4700 1 -- 172.21.15.190:0/1043350774 shutdown_connections 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.321+0000 7f95f5cc4700 1 -- 172.21.15.190:0/1043350774 wait complete. 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.322+0000 7f95f5cc4700 1 Processor -- start 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.322+0000 7f95f5cc4700 1 -- start start 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.322+0000 7f95f5cc4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 0x7f95f0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.322+0000 7f95f5cc4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f0060240 con 0x7f95f00f5410 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.322+0000 7f95ef7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 0x7f95f0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.322+0000 7f95ef7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 0x7f95f0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:53216/0 (socket says 172.21.15.190:53216) 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.322+0000 7f95ef7fe700 1 -- 172.21.15.190:0/2517350748 learned_addr learned my addr 172.21.15.190:0/2517350748 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:43.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.322+0000 7f95ef7fe700 1 -- 172.21.15.190:0/2517350748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d8005040 con 0x7f95f00f5410 2024-01-16T17:12:43.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.323+0000 7f95ef7fe700 1 --2- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 0x7f95f0109430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f95d8013510 tx=0x7f95d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:43.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.323+0000 7f95e7fff700 1 -- 172.21.15.190:0/2517350748 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95d8009320 con 0x7f95f00f5410 2024-01-16T17:12:43.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.323+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f0060440 con 0x7f95f00f5410 2024-01-16T17:12:43.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.323+0000 7f95e7fff700 1 -- 172.21.15.190:0/2517350748 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95d8013f00 con 0x7f95f00f5410 2024-01-16T17:12:43.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.323+0000 7f95e7fff700 1 -- 172.21.15.190:0/2517350748 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95d8014d90 con 0x7f95f00f5410 2024-01-16T17:12:43.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.323+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f0109ae0 con 0x7f95f00f5410 2024-01-16T17:12:43.307 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.324+0000 7f95e7fff700 1 -- 172.21.15.190:0/2517350748 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95d8014420 con 0x7f95f00f5410 2024-01-16T17:12:43.307 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.324+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f0045ad0 con 0x7f95f00f5410 2024-01-16T17:12:43.307 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.324+0000 7f95e7fff700 1 --2- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95dc02ff60 0x7f95dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:43.307 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.324+0000 7f95e7fff700 1 -- 172.21.15.190:0/2517350748 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95d8041000 con 0x7f95f00f5410 2024-01-16T17:12:43.307 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.324+0000 7f95eeffd700 1 --2- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95dc02ff60 0x7f95dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:43.307 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.324+0000 7f95eeffd700 1 --2- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95dc02ff60 0x7f95dc032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f95e00096a0 tx=0x7f95e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:43.310 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.328+0000 7f95e7fff700 1 -- 172.21.15.190:0/2517350748 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95d80095f0 con 0x7f95f00f5410 2024-01-16T17:12:43.516 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.535+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95f00fae60 con 0x7f95f00f5410 2024-01-16T17:12:43.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.536+0000 7f95e7fff700 1 -- 172.21.15.190:0/2517350748 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f95d801db70 con 0x7f95f00f5410 2024-01-16T17:12:43.517 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:43.518 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:43.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95dc02ff60 msgr2=0x7f95dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:43.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 --2- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95dc02ff60 0x7f95dc032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f95e00096a0 tx=0x7f95e0006b40).stop 2024-01-16T17:12:43.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 msgr2=0x7f95f0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:43.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 --2- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 0x7f95f0109430 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f95d8013510 tx=0x7f95d800b040).stop 2024-01-16T17:12:43.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 shutdown_connections 2024-01-16T17:12:43.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 --2- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95dc02ff60 0x7f95dc032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:43.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 --2- 172.21.15.190:0/2517350748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95f00f5410 0x7f95f0109430 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:43.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 >> 172.21.15.190:0/2517350748 conn(0x7f95f00f0b70 msgr2=0x7f95f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:43.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 shutdown_connections 2024-01-16T17:12:43.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:43.537+0000 7f95f5cc4700 1 -- 172.21.15.190:0/2517350748 wait complete. 2024-01-16T17:12:43.520 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:43.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:41.799358+0000 mgr.smithi154.sllivz (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:44.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:44.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:44.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:44 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:43.537604+0000 mon.smithi154 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.190:0/2517350748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:45.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:43.799719+0000 mgr.smithi154.sllivz (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:46.078 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.095+0000 7f12f0e91700 1 -- 172.21.15.190:0/1865438705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 msgr2=0x7f12ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:46.078 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.095+0000 7f12f0e91700 1 --2- 172.21.15.190:0/1865438705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 0x7f12ec0f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f12d4009fb0 tx=0x7f12d4009b30).stop 2024-01-16T17:12:46.078 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.095+0000 7f12f0e91700 1 -- 172.21.15.190:0/1865438705 shutdown_connections 2024-01-16T17:12:46.078 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.095+0000 7f12f0e91700 1 --2- 172.21.15.190:0/1865438705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 0x7f12ec0f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.095+0000 7f12f0e91700 1 -- 172.21.15.190:0/1865438705 >> 172.21.15.190:0/1865438705 conn(0x7f12ec0f0b70 msgr2=0x7f12ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.095+0000 7f12f0e91700 1 -- 172.21.15.190:0/1865438705 shutdown_connections 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.095+0000 7f12f0e91700 1 -- 172.21.15.190:0/1865438705 wait complete. 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.096+0000 7f12f0e91700 1 Processor -- start 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.096+0000 7f12f0e91700 1 -- start start 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.096+0000 7f12f0e91700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 0x7f12ec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.096+0000 7f12f0e91700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec105b20 con 0x7f12ec0f5410 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.096+0000 7f12ea59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 0x7f12ec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.096+0000 7f12ea59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 0x7f12ec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:53238/0 (socket says 172.21.15.190:53238) 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.096+0000 7f12ea59c700 1 -- 172.21.15.190:0/771054069 learned_addr learned my addr 172.21.15.190:0/771054069 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.097+0000 7f12ea59c700 1 -- 172.21.15.190:0/771054069 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d4005040 con 0x7f12ec0f5410 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.097+0000 7f12ea59c700 1 --2- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 0x7f12ec105630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f12d4013510 tx=0x7f12d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.097+0000 7f12e2ffd700 1 -- 172.21.15.190:0/771054069 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12d4009320 con 0x7f12ec0f5410 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.097+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ec105d20 con 0x7f12ec0f5410 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.097+0000 7f12e2ffd700 1 -- 172.21.15.190:0/771054069 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12d4013f00 con 0x7f12ec0f5410 2024-01-16T17:12:46.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.097+0000 7f12e2ffd700 1 -- 172.21.15.190:0/771054069 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12d4014d90 con 0x7f12ec0f5410 2024-01-16T17:12:46.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.097+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ec1061b0 con 0x7f12ec0f5410 2024-01-16T17:12:46.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.098+0000 7f12e2ffd700 1 -- 172.21.15.190:0/771054069 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12d40095f0 con 0x7f12ec0f5410 2024-01-16T17:12:46.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.098+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ec045ad0 con 0x7f12ec0f5410 2024-01-16T17:12:46.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.098+0000 7f12e2ffd700 1 --2- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f12d802ff60 0x7f12d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:46.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.098+0000 7f12e2ffd700 1 -- 172.21.15.190:0/771054069 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12d4041080 con 0x7f12ec0f5410 2024-01-16T17:12:46.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.098+0000 7f12e9d9b700 1 --2- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f12d802ff60 0x7f12d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:46.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.098+0000 7f12e9d9b700 1 --2- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f12d802ff60 0x7f12d8032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f12dc0096a0 tx=0x7f12dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:46.084 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.102+0000 7f12e2ffd700 1 -- 172.21.15.190:0/771054069 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12d4014420 con 0x7f12ec0f5410 2024-01-16T17:12:46.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.306+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12ec0fae60 con 0x7f12ec0f5410 2024-01-16T17:12:46.288 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.306+0000 7f12e2ffd700 1 -- 172.21.15.190:0/771054069 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f12d4010230 con 0x7f12ec0f5410 2024-01-16T17:12:46.288 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:46.288 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:46.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f12d802ff60 msgr2=0x7f12d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:46.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 --2- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f12d802ff60 0x7f12d8032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f12dc0096a0 tx=0x7f12dc006b40).stop 2024-01-16T17:12:46.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 msgr2=0x7f12ec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:46.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 --2- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 0x7f12ec105630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f12d4013510 tx=0x7f12d400b040).stop 2024-01-16T17:12:46.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 shutdown_connections 2024-01-16T17:12:46.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 --2- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f12d802ff60 0x7f12d8032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:46.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 --2- 172.21.15.190:0/771054069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12ec0f5410 0x7f12ec105630 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:46.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 >> 172.21.15.190:0/771054069 conn(0x7f12ec0f0b70 msgr2=0x7f12ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:46.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 shutdown_connections 2024-01-16T17:12:46.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:46.308+0000 7f12f0e91700 1 -- 172.21.15.190:0/771054069 wait complete. 2024-01-16T17:12:46.291 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:46.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:46 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:46.308594+0000 mon.smithi154 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.190:0/771054069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:47.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:47.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:47.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:45.800036+0000 mgr.smithi154.sllivz (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:48.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.833+0000 7f395f89b700 1 -- 172.21.15.190:0/3605707104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 msgr2=0x7f39580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:48.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.833+0000 7f395f89b700 1 --2- 172.21.15.190:0/3605707104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 0x7f39580f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f3948009fb0 tx=0x7f3948009b30).stop 2024-01-16T17:12:48.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.833+0000 7f395f89b700 1 -- 172.21.15.190:0/3605707104 shutdown_connections 2024-01-16T17:12:48.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.833+0000 7f395f89b700 1 --2- 172.21.15.190:0/3605707104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 0x7f39580f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:48.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.833+0000 7f395f89b700 1 -- 172.21.15.190:0/3605707104 >> 172.21.15.190:0/3605707104 conn(0x7f39580f0b70 msgr2=0x7f39580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:48.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.833+0000 7f395f89b700 1 -- 172.21.15.190:0/3605707104 shutdown_connections 2024-01-16T17:12:48.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.833+0000 7f395f89b700 1 -- 172.21.15.190:0/3605707104 wait complete. 2024-01-16T17:12:48.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.834+0000 7f395f89b700 1 Processor -- start 2024-01-16T17:12:48.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.834+0000 7f395f89b700 1 -- start start 2024-01-16T17:12:48.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.834+0000 7f395f89b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 0x7f3958109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:48.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.834+0000 7f395f89b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3958060240 con 0x7f39580f5410 2024-01-16T17:12:48.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.834+0000 7f395d637700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 0x7f3958109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:48.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.834+0000 7f395d637700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 0x7f3958109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:53258/0 (socket says 172.21.15.190:53258) 2024-01-16T17:12:48.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.834+0000 7f395d637700 1 -- 172.21.15.190:0/3752044007 learned_addr learned my addr 172.21.15.190:0/3752044007 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:48.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.835+0000 7f395d637700 1 -- 172.21.15.190:0/3752044007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3948005040 con 0x7f39580f5410 2024-01-16T17:12:48.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.835+0000 7f395d637700 1 --2- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 0x7f3958109430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f3948013510 tx=0x7f394800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:48.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.835+0000 7f394dffb700 1 -- 172.21.15.190:0/3752044007 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3948009320 con 0x7f39580f5410 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.835+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3958060440 con 0x7f39580f5410 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.835+0000 7f394dffb700 1 -- 172.21.15.190:0/3752044007 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3948013f00 con 0x7f39580f5410 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.835+0000 7f394dffb700 1 -- 172.21.15.190:0/3752044007 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3948014d90 con 0x7f39580f5410 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.835+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3958109ae0 con 0x7f39580f5410 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.836+0000 7f394dffb700 1 -- 172.21.15.190:0/3752044007 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39480263a0 con 0x7f39580f5410 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.836+0000 7f394dffb700 1 --2- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f394402ffb0 0x7f3944032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.836+0000 7f394dffb700 1 -- 172.21.15.190:0/3752044007 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39480410b0 con 0x7f39580f5410 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.836+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3958045ad0 con 0x7f39580f5410 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.836+0000 7f395ce36700 1 --2- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f394402ffb0 0x7f3944032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:48.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.837+0000 7f395ce36700 1 --2- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f394402ffb0 0x7f3944032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:48.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:48.841+0000 7f394dffb700 1 -- 172.21.15.190:0/3752044007 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3948012920 con 0x7f39580f5410 2024-01-16T17:12:49.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.047+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39580fae60 con 0x7f39580f5410 2024-01-16T17:12:49.029 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.048+0000 7f394dffb700 1 -- 172.21.15.190:0/3752044007 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f39580fae60 con 0x7f39580f5410 2024-01-16T17:12:49.029 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:49.029 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f394402ffb0 msgr2=0x7f3944032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 --2- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f394402ffb0 0x7f3944032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f39540096a0 tx=0x7f3954006b40).stop 2024-01-16T17:12:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 msgr2=0x7f3958109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 --2- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 0x7f3958109430 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f3948013510 tx=0x7f394800b040).stop 2024-01-16T17:12:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 shutdown_connections 2024-01-16T17:12:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 --2- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f394402ffb0 0x7f3944032470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:49.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 --2- 172.21.15.190:0/3752044007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39580f5410 0x7f3958109430 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:49.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 >> 172.21.15.190:0/3752044007 conn(0x7f39580f0b70 msgr2=0x7f39580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:49.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 shutdown_connections 2024-01-16T17:12:49.032 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:49.049+0000 7f395f89b700 1 -- 172.21.15.190:0/3752044007 wait complete. 2024-01-16T17:12:49.032 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:49.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:47.800385+0000 mgr.smithi154.sllivz (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:49.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:49.049497+0000 mon.smithi154 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.190:0/3752044007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:50.400 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:50.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.521+0000 7f9cd29e4700 1 -- 172.21.15.190:0/1565855726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 msgr2=0x7f9ccc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.521+0000 7f9cd29e4700 1 --2- 172.21.15.190:0/1565855726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 0x7f9ccc0f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f9cbc009fb0 tx=0x7f9cbc009b30).stop 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.522+0000 7f9cd29e4700 1 -- 172.21.15.190:0/1565855726 shutdown_connections 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.522+0000 7f9cd29e4700 1 --2- 172.21.15.190:0/1565855726 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 0x7f9ccc0f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.522+0000 7f9cd29e4700 1 -- 172.21.15.190:0/1565855726 >> 172.21.15.190:0/1565855726 conn(0x7f9ccc0f0b70 msgr2=0x7f9ccc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.522+0000 7f9cd29e4700 1 -- 172.21.15.190:0/1565855726 shutdown_connections 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.522+0000 7f9cd29e4700 1 -- 172.21.15.190:0/1565855726 wait complete. 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.522+0000 7f9cd29e4700 1 Processor -- start 2024-01-16T17:12:51.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.522+0000 7f9cd29e4700 1 -- start start 2024-01-16T17:12:51.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.523+0000 7f9cd29e4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 0x7f9ccc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:51.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.523+0000 7f9cd29e4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc105b20 con 0x7f9ccc0f5410 2024-01-16T17:12:51.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.523+0000 7f9ccbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 0x7f9ccc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:51.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.523+0000 7f9ccbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 0x7f9ccc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:53278/0 (socket says 172.21.15.190:53278) 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.523+0000 7f9ccbfff700 1 -- 172.21.15.190:0/141584483 learned_addr learned my addr 172.21.15.190:0/141584483 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.523+0000 7f9ccbfff700 1 -- 172.21.15.190:0/141584483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cbc005040 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.523+0000 7f9ccbfff700 1 --2- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 0x7f9ccc105630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9cbc013510 tx=0x7f9cbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.524+0000 7f9cc8ff9700 1 -- 172.21.15.190:0/141584483 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cbc009320 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.524+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ccc105d20 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.524+0000 7f9cc8ff9700 1 -- 172.21.15.190:0/141584483 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cbc013f00 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.524+0000 7f9cc8ff9700 1 -- 172.21.15.190:0/141584483 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cbc014d90 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.524+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ccc1061b0 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.524+0000 7f9cc8ff9700 1 -- 172.21.15.190:0/141584483 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cbc014420 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.524+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ccc045ad0 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.525+0000 7f9cc8ff9700 1 --2- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cb402ff60 0x7f9cb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.525+0000 7f9cc8ff9700 1 -- 172.21.15.190:0/141584483 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cbc0410c0 con 0x7f9ccc0f5410 2024-01-16T17:12:51.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.525+0000 7f9ccb7fe700 1 --2- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cb402ff60 0x7f9cb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:51.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.525+0000 7f9ccb7fe700 1 --2- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cb402ff60 0x7f9cb4032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9cc00096a0 tx=0x7f9cc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.529+0000 7f9cc8ff9700 1 -- 172.21.15.190:0/141584483 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cbc0095f0 con 0x7f9ccc0f5410 2024-01-16T17:12:51.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.737+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ccc0fae60 con 0x7f9ccc0f5410 2024-01-16T17:12:51.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.738+0000 7f9cc8ff9700 1 -- 172.21.15.190:0/141584483 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9cbc01db70 con 0x7f9ccc0f5410 2024-01-16T17:12:51.718 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:51.718 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:51.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cb402ff60 msgr2=0x7f9cb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:51.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 --2- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cb402ff60 0x7f9cb4032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9cc00096a0 tx=0x7f9cc0006b40).stop 2024-01-16T17:12:51.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 msgr2=0x7f9ccc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:51.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 --2- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 0x7f9ccc105630 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f9cbc013510 tx=0x7f9cbc00b040).stop 2024-01-16T17:12:51.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 shutdown_connections 2024-01-16T17:12:51.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 --2- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cb402ff60 0x7f9cb4032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:51.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 --2- 172.21.15.190:0/141584483 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ccc0f5410 0x7f9ccc105630 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:51.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 >> 172.21.15.190:0/141584483 conn(0x7f9ccc0f0b70 msgr2=0x7f9ccc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:51.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 shutdown_connections 2024-01-16T17:12:51.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:51.739+0000 7f9cd29e4700 1 -- 172.21.15.190:0/141584483 wait complete. 2024-01-16T17:12:51.721 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:51.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:49.800730+0000 mgr.smithi154.sllivz (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:52.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:52 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:51.738727+0000 mon.smithi154 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.190:0/141584483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:53.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:53.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:53.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:51.801011+0000 mgr.smithi154.sllivz (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:54.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.211+0000 7f11b386d700 1 -- 172.21.15.190:0/981130952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 msgr2=0x7f11ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.211+0000 7f11b386d700 1 --2- 172.21.15.190:0/981130952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 0x7f11ac0f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f119c009fb0 tx=0x7f119c009b30).stop 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.211+0000 7f11b386d700 1 -- 172.21.15.190:0/981130952 shutdown_connections 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.211+0000 7f11b386d700 1 --2- 172.21.15.190:0/981130952 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 0x7f11ac0f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.211+0000 7f11b386d700 1 -- 172.21.15.190:0/981130952 >> 172.21.15.190:0/981130952 conn(0x7f11ac0f0b70 msgr2=0x7f11ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.211+0000 7f11b386d700 1 -- 172.21.15.190:0/981130952 shutdown_connections 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.211+0000 7f11b386d700 1 -- 172.21.15.190:0/981130952 wait complete. 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.212+0000 7f11b386d700 1 Processor -- start 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.212+0000 7f11b386d700 1 -- start start 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.212+0000 7f11b386d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 0x7f11ac106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:54.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.212+0000 7f11b386d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac106c10 con 0x7f11ac0f5410 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.212+0000 7f11b1609700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 0x7f11ac106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.212+0000 7f11b1609700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 0x7f11ac106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52592/0 (socket says 172.21.15.190:52592) 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.212+0000 7f11b1609700 1 -- 172.21.15.190:0/1647978637 learned_addr learned my addr 172.21.15.190:0/1647978637 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.213+0000 7f11b1609700 1 -- 172.21.15.190:0/1647978637 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c005040 con 0x7f11ac0f5410 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.213+0000 7f11b1609700 1 --2- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 0x7f11ac106720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f119c013510 tx=0x7f119c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.213+0000 7f11a1ffb700 1 -- 172.21.15.190:0/1647978637 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f119c009320 con 0x7f11ac0f5410 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.213+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac106e10 con 0x7f11ac0f5410 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.213+0000 7f11a1ffb700 1 -- 172.21.15.190:0/1647978637 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f119c013f00 con 0x7f11ac0f5410 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.213+0000 7f11a1ffb700 1 -- 172.21.15.190:0/1647978637 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f119c014d90 con 0x7f11ac0f5410 2024-01-16T17:12:54.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.213+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac1072a0 con 0x7f11ac0f5410 2024-01-16T17:12:54.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.214+0000 7f11a1ffb700 1 -- 172.21.15.190:0/1647978637 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f119c0263a0 con 0x7f11ac0f5410 2024-01-16T17:12:54.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.214+0000 7f11a1ffb700 1 --2- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f119802ffb0 0x7f1198032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:54.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.214+0000 7f11a1ffb700 1 -- 172.21.15.190:0/1647978637 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f119c0410b0 con 0x7f11ac0f5410 2024-01-16T17:12:54.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.214+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac045ad0 con 0x7f11ac0f5410 2024-01-16T17:12:54.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.214+0000 7f11b0e08700 1 --2- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f119802ffb0 0x7f1198032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:54.197 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.215+0000 7f11b0e08700 1 --2- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f119802ffb0 0x7f1198032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f11a80096a0 tx=0x7f11a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:54.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.219+0000 7f11a1ffb700 1 -- 172.21.15.190:0/1647978637 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f119c012920 con 0x7f11ac0f5410 2024-01-16T17:12:54.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.424+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11ac107fc0 con 0x7f11ac0f5410 2024-01-16T17:12:54.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.425+0000 7f11a1ffb700 1 -- 172.21.15.190:0/1647978637 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f119c02b020 con 0x7f11ac0f5410 2024-01-16T17:12:54.405 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:54.406 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:54.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f119802ffb0 msgr2=0x7f1198032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:54.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 --2- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f119802ffb0 0x7f1198032470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f11a80096a0 tx=0x7f11a8006b40).stop 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 msgr2=0x7f11ac106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 --2- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 0x7f11ac106720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f119c013510 tx=0x7f119c00b040).stop 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 shutdown_connections 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 --2- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f119802ffb0 0x7f1198032470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 --2- 172.21.15.190:0/1647978637 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11ac0f5410 0x7f11ac106720 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 >> 172.21.15.190:0/1647978637 conn(0x7f11ac0f0b70 msgr2=0x7f11ac05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 shutdown_connections 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:54.426+0000 7f11b386d700 1 -- 172.21.15.190:0/1647978637 wait complete. 2024-01-16T17:12:54.408 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:54.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:54 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:54.425772+0000 mon.smithi154 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.190:0/1647978637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:55.774 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:55.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:55.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:53.801361+0000 mgr.smithi154.sllivz (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:56.897 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.915+0000 7f5d54f12700 1 -- 172.21.15.190:0/3047590307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 msgr2=0x7f5d500f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:56.897 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.915+0000 7f5d54f12700 1 --2- 172.21.15.190:0/3047590307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 0x7f5d500f91b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f5d38009fb0 tx=0x7f5d38009b30).stop 2024-01-16T17:12:56.897 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 -- 172.21.15.190:0/3047590307 shutdown_connections 2024-01-16T17:12:56.897 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 --2- 172.21.15.190:0/3047590307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 0x7f5d500f91b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:56.897 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 -- 172.21.15.190:0/3047590307 >> 172.21.15.190:0/3047590307 conn(0x7f5d500f0b70 msgr2=0x7f5d500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:56.897 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 -- 172.21.15.190:0/3047590307 shutdown_connections 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 -- 172.21.15.190:0/3047590307 wait complete. 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 Processor -- start 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 -- start start 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 0x7f5d500f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.916+0000 7f5d54f12700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d500f50b0 con 0x7f5d500601c0 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d4e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 0x7f5d500f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d4e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 0x7f5d500f6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52600/0 (socket says 172.21.15.190:52600) 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d4e59c700 1 -- 172.21.15.190:0/2851825181 learned_addr learned my addr 172.21.15.190:0/2851825181 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d4e59c700 1 -- 172.21.15.190:0/2851825181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d38005040 con 0x7f5d500601c0 2024-01-16T17:12:56.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d4e59c700 1 --2- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 0x7f5d500f6c30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5d38009830 tx=0x7f5d3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:56.899 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d46ffd700 1 -- 172.21.15.190:0/2851825181 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d38009320 con 0x7f5d500601c0 2024-01-16T17:12:56.899 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d500f5250 con 0x7f5d500601c0 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d46ffd700 1 -- 172.21.15.190:0/2851825181 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d380136c0 con 0x7f5d500601c0 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d46ffd700 1 -- 172.21.15.190:0/2851825181 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d3801ee90 con 0x7f5d500601c0 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.917+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d500f56e0 con 0x7f5d500601c0 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.918+0000 7f5d46ffd700 1 -- 172.21.15.190:0/2851825181 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d3801e420 con 0x7f5d500601c0 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.918+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d50059100 con 0x7f5d500601c0 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.918+0000 7f5d46ffd700 1 --2- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.918+0000 7f5d46ffd700 1 -- 172.21.15.190:0/2851825181 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d38040fa0 con 0x7f5d500601c0 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.918+0000 7f5d4dd9b700 1 --2- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.919+0000 7f5d4dd9b700 1 --2- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f5d400096a0 tx=0x7f5d40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:56.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:56.922+0000 7f5d46ffd700 1 -- 172.21.15.190:0/2851825181 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d380095f0 con 0x7f5d500601c0 2024-01-16T17:12:56.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:55.926569+0000 mon.smithi154 (mon.0) 150 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:12:56.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:55.928409+0000 mon.smithi154 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:12:57.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.129+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d500fbf50 con 0x7f5d500601c0 2024-01-16T17:12:57.110 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.129+0000 7f5d46ffd700 1 -- 172.21.15.190:0/2851825181 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d3802c2d0 con 0x7f5d500601c0 2024-01-16T17:12:57.110 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:57.110 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3c02ff10 msgr2=0x7f5d3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 --2- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f5d400096a0 tx=0x7f5d40006b40).stop 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 msgr2=0x7f5d500f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 --2- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 0x7f5d500f6c30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f5d38009830 tx=0x7f5d3800b040).stop 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 shutdown_connections 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 --2- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3c02ff10 0x7f5d3c0323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 --2- 172.21.15.190:0/2851825181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d500601c0 0x7f5d500f6c30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 >> 172.21.15.190:0/2851825181 conn(0x7f5d500f0b70 msgr2=0x7f5d500f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:57.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 shutdown_connections 2024-01-16T17:12:57.113 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:57.131+0000 7f5d54f12700 1 -- 172.21.15.190:0/2851825181 wait complete. 2024-01-16T17:12:57.113 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:57.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:55.801611+0000 mgr.smithi154.sllivz (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:12:57.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:57.130368+0000 mon.smithi154 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.190:0/2851825181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:12:58.484 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:12:58.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:12:59.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.627+0000 7fc4e0946700 1 -- 172.21.15.190:0/2357094360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 msgr2=0x7fc4dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:59.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.627+0000 7fc4e0946700 1 --2- 172.21.15.190:0/2357094360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 0x7fc4dc0f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc4c4009fb0 tx=0x7fc4c4009b30).stop 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.627+0000 7fc4e0946700 1 -- 172.21.15.190:0/2357094360 shutdown_connections 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.627+0000 7fc4e0946700 1 --2- 172.21.15.190:0/2357094360 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 0x7fc4dc0f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.627+0000 7fc4e0946700 1 -- 172.21.15.190:0/2357094360 >> 172.21.15.190:0/2357094360 conn(0x7fc4dc0f0b70 msgr2=0x7fc4dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.627+0000 7fc4e0946700 1 -- 172.21.15.190:0/2357094360 shutdown_connections 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.627+0000 7fc4e0946700 1 -- 172.21.15.190:0/2357094360 wait complete. 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.628+0000 7fc4e0946700 1 Processor -- start 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.628+0000 7fc4e0946700 1 -- start start 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.628+0000 7fc4e0946700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 0x7fc4dc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.628+0000 7fc4e0946700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc105af0 con 0x7fc4dc0f5410 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.628+0000 7fc4da59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 0x7fc4dc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.628+0000 7fc4da59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 0x7fc4dc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52616/0 (socket says 172.21.15.190:52616) 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.628+0000 7fc4da59c700 1 -- 172.21.15.190:0/1666108912 learned_addr learned my addr 172.21.15.190:0/1666108912 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.628+0000 7fc4da59c700 1 -- 172.21.15.190:0/1666108912 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4005040 con 0x7fc4dc0f5410 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.629+0000 7fc4da59c700 1 --2- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 0x7fc4dc105600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc4c4013510 tx=0x7fc4c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:59.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.629+0000 7fc4d2ffd700 1 -- 172.21.15.190:0/1666108912 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4c4009320 con 0x7fc4dc0f5410 2024-01-16T17:12:59.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.629+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4dc105cf0 con 0x7fc4dc0f5410 2024-01-16T17:12:59.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.629+0000 7fc4d2ffd700 1 -- 172.21.15.190:0/1666108912 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4c4013f00 con 0x7fc4dc0f5410 2024-01-16T17:12:59.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.629+0000 7fc4d2ffd700 1 -- 172.21.15.190:0/1666108912 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4c4014d90 con 0x7fc4dc0f5410 2024-01-16T17:12:59.611 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.629+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc106180 con 0x7fc4dc0f5410 2024-01-16T17:12:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.630+0000 7fc4d2ffd700 1 -- 172.21.15.190:0/1666108912 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc4c40095f0 con 0x7fc4dc0f5410 2024-01-16T17:12:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.630+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc045ad0 con 0x7fc4dc0f5410 2024-01-16T17:12:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.630+0000 7fc4d2ffd700 1 --2- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc4c802ff60 0x7fc4c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:12:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.630+0000 7fc4d2ffd700 1 -- 172.21.15.190:0/1666108912 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4c40410c0 con 0x7fc4dc0f5410 2024-01-16T17:12:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.630+0000 7fc4d9d9b700 1 --2- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc4c802ff60 0x7fc4c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:12:59.612 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.630+0000 7fc4d9d9b700 1 --2- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc4c802ff60 0x7fc4c8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc4cc0096a0 tx=0x7fc4cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:12:59.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.634+0000 7fc4d2ffd700 1 -- 172.21.15.190:0/1666108912 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4c401d7f0 con 0x7fc4dc0f5410 2024-01-16T17:12:59.825 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.845+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4dc0fae60 con 0x7fc4dc0f5410 2024-01-16T17:12:59.826 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.846+0000 7fc4d2ffd700 1 -- 172.21.15.190:0/1666108912 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc4c4010230 con 0x7fc4dc0f5410 2024-01-16T17:12:59.826 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:12:59.826 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:12:59.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc4c802ff60 msgr2=0x7fc4c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:59.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 --2- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc4c802ff60 0x7fc4c8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fc4cc0096a0 tx=0x7fc4cc006b40).stop 2024-01-16T17:12:59.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 msgr2=0x7fc4dc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:12:59.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 --2- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 0x7fc4dc105600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fc4c4013510 tx=0x7fc4c400b040).stop 2024-01-16T17:12:59.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 shutdown_connections 2024-01-16T17:12:59.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 --2- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc4c802ff60 0x7fc4c8032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:59.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 --2- 172.21.15.190:0/1666108912 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4dc0f5410 0x7fc4dc105600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:12:59.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 >> 172.21.15.190:0/1666108912 conn(0x7fc4dc0f0b70 msgr2=0x7fc4dc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:12:59.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 shutdown_connections 2024-01-16T17:12:59.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:12:59.847+0000 7fc4e0946700 1 -- 172.21.15.190:0/1666108912 wait complete. 2024-01-16T17:12:59.829 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:12:59.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:12:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:57.801915+0000 mgr.smithi154.sllivz (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:00.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:00 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:12:59.846686+0000 mon.smithi154 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.190:0/1666108912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:01.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:01.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:01.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:12:59.802289+0000 mgr.smithi154.sllivz (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.369+0000 7feda9c4b700 1 -- 172.21.15.190:0/2349669538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 msgr2=0x7feda40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.369+0000 7feda9c4b700 1 --2- 172.21.15.190:0/2349669538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 0x7feda40f55c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fed98009fb0 tx=0x7fed98009b30).stop 2024-01-16T17:13:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.369+0000 7feda9c4b700 1 -- 172.21.15.190:0/2349669538 shutdown_connections 2024-01-16T17:13:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.369+0000 7feda9c4b700 1 --2- 172.21.15.190:0/2349669538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 0x7feda40f55c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.369+0000 7feda9c4b700 1 -- 172.21.15.190:0/2349669538 >> 172.21.15.190:0/2349669538 conn(0x7feda40f09a0 msgr2=0x7feda40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.369+0000 7feda9c4b700 1 -- 172.21.15.190:0/2349669538 shutdown_connections 2024-01-16T17:13:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.369+0000 7feda9c4b700 1 -- 172.21.15.190:0/2349669538 wait complete. 2024-01-16T17:13:02.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.369+0000 7feda9c4b700 1 Processor -- start 2024-01-16T17:13:02.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.370+0000 7feda9c4b700 1 -- start start 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.370+0000 7feda9c4b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 0x7feda41064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.370+0000 7feda9c4b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda41069d0 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.370+0000 7feda8c49700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 0x7feda41064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.370+0000 7feda8c49700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 0x7feda41064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58822/0 (socket says 172.21.15.190:58822) 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.370+0000 7feda8c49700 1 -- 172.21.15.190:0/1473881 learned_addr learned my addr 172.21.15.190:0/1473881 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.370+0000 7feda8c49700 1 -- 172.21.15.190:0/1473881 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed98005040 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.371+0000 7feda8c49700 1 --2- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 0x7feda41064e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fed98013510 tx=0x7fed9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.371+0000 7feda17fa700 1 -- 172.21.15.190:0/1473881 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed98009320 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.371+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda4106bd0 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.371+0000 7feda17fa700 1 -- 172.21.15.190:0/1473881 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed98013f00 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.371+0000 7feda17fa700 1 -- 172.21.15.190:0/1473881 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed98014dc0 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.371+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda4107060 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.372+0000 7feda17fa700 1 -- 172.21.15.190:0/1473881 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed980095f0 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.372+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda4045ad0 con 0x7feda40f51a0 2024-01-16T17:13:02.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.372+0000 7feda17fa700 1 --2- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed9402ff60 0x7fed94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:02.354 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.372+0000 7feda17fa700 1 -- 172.21.15.190:0/1473881 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed980410e0 con 0x7feda40f51a0 2024-01-16T17:13:02.354 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.372+0000 7feda3fff700 1 --2- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed9402ff60 0x7fed94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:02.354 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.372+0000 7feda3fff700 1 --2- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed9402ff60 0x7fed94032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fed900096a0 tx=0x7fed90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:02.356 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.376+0000 7feda17fa700 1 -- 172.21.15.190:0/1473881 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed98014420 con 0x7feda40f51a0 2024-01-16T17:13:02.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.583+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feda4107eb0 con 0x7feda40f51a0 2024-01-16T17:13:02.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.584+0000 7feda17fa700 1 -- 172.21.15.190:0/1473881 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed9801c920 con 0x7feda40f51a0 2024-01-16T17:13:02.566 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:02.566 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed9402ff60 msgr2=0x7fed94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 --2- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed9402ff60 0x7fed94032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fed900096a0 tx=0x7fed90006b40).stop 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 msgr2=0x7feda41064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 --2- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 0x7feda41064e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fed98013510 tx=0x7fed9800b040).stop 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 shutdown_connections 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 --2- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed9402ff60 0x7fed94032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 --2- 172.21.15.190:0/1473881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feda40f51a0 0x7feda41064e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 >> 172.21.15.190:0/1473881 conn(0x7feda40f09a0 msgr2=0x7feda405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 shutdown_connections 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:02.585+0000 7feda9c4b700 1 -- 172.21.15.190:0/1473881 wait complete. 2024-01-16T17:13:02.567 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:03.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:01.802653+0000 mgr.smithi154.sllivz (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:03.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:02.584545+0000 mon.smithi154 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.190:0/1473881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:03.986 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:03.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.118+0000 7f66c2e10700 1 -- 172.21.15.190:0/3932195785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 msgr2=0x7f66bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.118+0000 7f66c2e10700 1 --2- 172.21.15.190:0/3932195785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 0x7f66bc0f55c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f66ac009fb0 tx=0x7f66ac009b30).stop 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.118+0000 7f66c2e10700 1 -- 172.21.15.190:0/3932195785 shutdown_connections 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.118+0000 7f66c2e10700 1 --2- 172.21.15.190:0/3932195785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 0x7f66bc0f55c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.118+0000 7f66c2e10700 1 -- 172.21.15.190:0/3932195785 >> 172.21.15.190:0/3932195785 conn(0x7f66bc0f09a0 msgr2=0x7f66bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.119+0000 7f66c2e10700 1 -- 172.21.15.190:0/3932195785 shutdown_connections 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.119+0000 7f66c2e10700 1 -- 172.21.15.190:0/3932195785 wait complete. 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.119+0000 7f66c2e10700 1 Processor -- start 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.119+0000 7f66c2e10700 1 -- start start 2024-01-16T17:13:05.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.119+0000 7f66c2e10700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 0x7f66bc1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.119+0000 7f66c2e10700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66bc1058a0 con 0x7f66bc0f51a0 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66c1e0e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 0x7f66bc1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66c1e0e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 0x7f66bc1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58844/0 (socket says 172.21.15.190:58844) 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66c1e0e700 1 -- 172.21.15.190:0/1079805003 learned_addr learned my addr 172.21.15.190:0/1079805003 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66c1e0e700 1 -- 172.21.15.190:0/1079805003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac005040 con 0x7f66bc0f51a0 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66c1e0e700 1 --2- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 0x7f66bc1053b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f66ac013510 tx=0x7f66ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66b27fc700 1 -- 172.21.15.190:0/1079805003 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66ac009320 con 0x7f66bc0f51a0 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66bc105aa0 con 0x7f66bc0f51a0 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66b27fc700 1 -- 172.21.15.190:0/1079805003 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66ac013f00 con 0x7f66bc0f51a0 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66b27fc700 1 -- 172.21.15.190:0/1079805003 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66ac014dc0 con 0x7f66bc0f51a0 2024-01-16T17:13:05.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.120+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66bc105f30 con 0x7f66bc0f51a0 2024-01-16T17:13:05.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.121+0000 7f66b27fc700 1 -- 172.21.15.190:0/1079805003 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66ac0095f0 con 0x7f66bc0f51a0 2024-01-16T17:13:05.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.121+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66bc045ad0 con 0x7f66bc0f51a0 2024-01-16T17:13:05.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.121+0000 7f66b27fc700 1 --2- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f66a802ff60 0x7f66a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:05.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.121+0000 7f66b27fc700 1 -- 172.21.15.190:0/1079805003 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66ac041040 con 0x7f66bc0f51a0 2024-01-16T17:13:05.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.122+0000 7f66c160d700 1 --2- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f66a802ff60 0x7f66a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:05.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.122+0000 7f66c160d700 1 --2- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f66a802ff60 0x7f66a8032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f66b80096a0 tx=0x7f66b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:05.106 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.126+0000 7f66b27fc700 1 -- 172.21.15.190:0/1079805003 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66ac014420 con 0x7f66bc0f51a0 2024-01-16T17:13:05.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.334+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66bc106de0 con 0x7f66bc0f51a0 2024-01-16T17:13:05.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.334+0000 7f66b27fc700 1 -- 172.21.15.190:0/1079805003 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f66ac010230 con 0x7f66bc0f51a0 2024-01-16T17:13:05.315 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:05.315 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f66a802ff60 msgr2=0x7f66a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 --2- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f66a802ff60 0x7f66a8032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f66b80096a0 tx=0x7f66b8006b40).stop 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 msgr2=0x7f66bc1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 --2- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 0x7f66bc1053b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f66ac013510 tx=0x7f66ac00b040).stop 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 shutdown_connections 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 --2- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f66a802ff60 0x7f66a8032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 --2- 172.21.15.190:0/1079805003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66bc0f51a0 0x7f66bc1053b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 >> 172.21.15.190:0/1079805003 conn(0x7f66bc0f09a0 msgr2=0x7f66bc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 shutdown_connections 2024-01-16T17:13:05.317 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:05.336+0000 7f66c2e10700 1 -- 172.21.15.190:0/1079805003 wait complete. 2024-01-16T17:13:05.318 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:05.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:03.803009+0000 mgr.smithi154.sllivz (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:05.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:05.335403+0000 mon.smithi154 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.190:0/1079805003' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:06.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:06.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:07.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.874+0000 7f6b3dde0700 1 -- 172.21.15.190:0/272178166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 msgr2=0x7f6b380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.874+0000 7f6b3dde0700 1 --2- 172.21.15.190:0/272178166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 0x7f6b380f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f6b20009fb0 tx=0x7f6b20009b30).stop 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.874+0000 7f6b3dde0700 1 -- 172.21.15.190:0/272178166 shutdown_connections 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.874+0000 7f6b3dde0700 1 --2- 172.21.15.190:0/272178166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 0x7f6b380f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.874+0000 7f6b3dde0700 1 -- 172.21.15.190:0/272178166 >> 172.21.15.190:0/272178166 conn(0x7f6b380f0b70 msgr2=0x7f6b380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.874+0000 7f6b3dde0700 1 -- 172.21.15.190:0/272178166 shutdown_connections 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.874+0000 7f6b3dde0700 1 -- 172.21.15.190:0/272178166 wait complete. 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.875+0000 7f6b3dde0700 1 Processor -- start 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.875+0000 7f6b3dde0700 1 -- start start 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.875+0000 7f6b3dde0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 0x7f6b38109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.875+0000 7f6b3dde0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b38060220 con 0x7f6b380f5410 2024-01-16T17:13:07.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.875+0000 7f6b377fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 0x7f6b38109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.875+0000 7f6b377fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 0x7f6b38109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58864/0 (socket says 172.21.15.190:58864) 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.875+0000 7f6b377fe700 1 -- 172.21.15.190:0/30999666 learned_addr learned my addr 172.21.15.190:0/30999666 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.876+0000 7f6b377fe700 1 -- 172.21.15.190:0/30999666 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b20005040 con 0x7f6b380f5410 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.876+0000 7f6b377fe700 1 --2- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 0x7f6b38109450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f6b20013510 tx=0x7f6b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.876+0000 7f6b2ffff700 1 -- 172.21.15.190:0/30999666 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b20009320 con 0x7f6b380f5410 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.876+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b38060420 con 0x7f6b380f5410 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.876+0000 7f6b2ffff700 1 -- 172.21.15.190:0/30999666 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b20013f00 con 0x7f6b380f5410 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.876+0000 7f6b2ffff700 1 -- 172.21.15.190:0/30999666 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b20014d90 con 0x7f6b380f5410 2024-01-16T17:13:07.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.876+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b38109b00 con 0x7f6b380f5410 2024-01-16T17:13:07.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.877+0000 7f6b2ffff700 1 -- 172.21.15.190:0/30999666 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b20014420 con 0x7f6b380f5410 2024-01-16T17:13:07.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.877+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b38059100 con 0x7f6b380f5410 2024-01-16T17:13:07.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.877+0000 7f6b2ffff700 1 --2- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b2402ff60 0x7f6b24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:07.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.877+0000 7f6b2ffff700 1 -- 172.21.15.190:0/30999666 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b20041050 con 0x7f6b380f5410 2024-01-16T17:13:07.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.877+0000 7f6b36ffd700 1 --2- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b2402ff60 0x7f6b24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:07.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.878+0000 7f6b36ffd700 1 --2- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b2402ff60 0x7f6b24032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6b280096a0 tx=0x7f6b28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:07.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:07.882+0000 7f6b2ffff700 1 -- 172.21.15.190:0/30999666 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b200095f0 con 0x7f6b380f5410 2024-01-16T17:13:07.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:05.803330+0000 mgr.smithi154.sllivz (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:08.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.092+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b38045ad0 con 0x7f6b380f5410 2024-01-16T17:13:08.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.093+0000 7f6b2ffff700 1 -- 172.21.15.190:0/30999666 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6b2001db70 con 0x7f6b380f5410 2024-01-16T17:13:08.073 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:08.073 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b2402ff60 msgr2=0x7f6b24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 --2- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b2402ff60 0x7f6b24032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6b280096a0 tx=0x7f6b28006b40).stop 2024-01-16T17:13:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 msgr2=0x7f6b38109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 --2- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 0x7f6b38109450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f6b20013510 tx=0x7f6b2000b040).stop 2024-01-16T17:13:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 shutdown_connections 2024-01-16T17:13:08.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 --2- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6b2402ff60 0x7f6b24032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:08.076 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 --2- 172.21.15.190:0/30999666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b380f5410 0x7f6b38109450 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:08.076 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 >> 172.21.15.190:0/30999666 conn(0x7f6b380f0b70 msgr2=0x7f6b380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:08.076 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 shutdown_connections 2024-01-16T17:13:08.076 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:08.094+0000 7f6b3dde0700 1 -- 172.21.15.190:0/30999666 wait complete. 2024-01-16T17:13:08.076 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:08.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:08.093665+0000 mon.smithi154 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.190:0/30999666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:09.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:09.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:09.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:07.803599+0000 mgr.smithi154.sllivz (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:10.559 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.577+0000 7eff7a387700 1 -- 172.21.15.190:0/1561584832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 msgr2=0x7eff740f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:10.559 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.577+0000 7eff7a387700 1 --2- 172.21.15.190:0/1561584832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 0x7eff740f6df0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7eff64009fb0 tx=0x7eff64009b30).stop 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.577+0000 7eff7a387700 1 -- 172.21.15.190:0/1561584832 shutdown_connections 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.577+0000 7eff7a387700 1 --2- 172.21.15.190:0/1561584832 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 0x7eff740f6df0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.577+0000 7eff7a387700 1 -- 172.21.15.190:0/1561584832 >> 172.21.15.190:0/1561584832 conn(0x7eff740f02e0 msgr2=0x7eff740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.577+0000 7eff7a387700 1 -- 172.21.15.190:0/1561584832 shutdown_connections 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.577+0000 7eff7a387700 1 -- 172.21.15.190:0/1561584832 wait complete. 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.578+0000 7eff7a387700 1 Processor -- start 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.578+0000 7eff7a387700 1 -- start start 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.578+0000 7eff7a387700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 0x7eff741093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.578+0000 7eff7a387700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff740f5250 con 0x7eff740f69d0 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.578+0000 7eff73fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 0x7eff741093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.578+0000 7eff73fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 0x7eff741093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58880/0 (socket says 172.21.15.190:58880) 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.578+0000 7eff73fff700 1 -- 172.21.15.190:0/2878079539 learned_addr learned my addr 172.21.15.190:0/2878079539 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:10.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.578+0000 7eff73fff700 1 -- 172.21.15.190:0/2878079539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff64005040 con 0x7eff740f69d0 2024-01-16T17:13:10.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.579+0000 7eff73fff700 1 --2- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 0x7eff741093c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7eff64013510 tx=0x7eff6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:10.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.579+0000 7eff70ff9700 1 -- 172.21.15.190:0/2878079539 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff64009320 con 0x7eff740f69d0 2024-01-16T17:13:10.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.579+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff740f5450 con 0x7eff740f69d0 2024-01-16T17:13:10.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.579+0000 7eff70ff9700 1 -- 172.21.15.190:0/2878079539 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff64013f00 con 0x7eff740f69d0 2024-01-16T17:13:10.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.579+0000 7eff70ff9700 1 -- 172.21.15.190:0/2878079539 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff64014d90 con 0x7eff740f69d0 2024-01-16T17:13:10.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.579+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff74109a70 con 0x7eff740f69d0 2024-01-16T17:13:10.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.580+0000 7eff70ff9700 1 -- 172.21.15.190:0/2878079539 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff640095f0 con 0x7eff740f69d0 2024-01-16T17:13:10.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.580+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff74059100 con 0x7eff740f69d0 2024-01-16T17:13:10.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.580+0000 7eff70ff9700 1 --2- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7eff5c02ff60 0x7eff5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:10.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.580+0000 7eff70ff9700 1 -- 172.21.15.190:0/2878079539 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff64041000 con 0x7eff740f69d0 2024-01-16T17:13:10.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.580+0000 7eff737fe700 1 --2- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7eff5c02ff60 0x7eff5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:10.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.581+0000 7eff737fe700 1 --2- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7eff5c02ff60 0x7eff5c032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7eff680096a0 tx=0x7eff68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:10.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.584+0000 7eff70ff9700 1 -- 172.21.15.190:0/2878079539 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff64014420 con 0x7eff740f69d0 2024-01-16T17:13:10.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.791+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff74045ad0 con 0x7eff740f69d0 2024-01-16T17:13:10.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.791+0000 7eff70ff9700 1 -- 172.21.15.190:0/2878079539 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7eff64010230 con 0x7eff740f69d0 2024-01-16T17:13:10.772 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:10.772 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:10.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7eff5c02ff60 msgr2=0x7eff5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:10.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 --2- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7eff5c02ff60 0x7eff5c032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7eff680096a0 tx=0x7eff68006b40).stop 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 msgr2=0x7eff741093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 --2- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 0x7eff741093c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7eff64013510 tx=0x7eff6400b040).stop 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 shutdown_connections 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 --2- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7eff5c02ff60 0x7eff5c032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 --2- 172.21.15.190:0/2878079539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff740f69d0 0x7eff741093c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 >> 172.21.15.190:0/2878079539 conn(0x7eff740f02e0 msgr2=0x7eff740f0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 shutdown_connections 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:10.793+0000 7eff7a387700 1 -- 172.21.15.190:0/2878079539 wait complete. 2024-01-16T17:13:10.774 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:11.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:09.803933+0000 mgr.smithi154.sllivz (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:11.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:10.792502+0000 mon.smithi154 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.190:0/2878079539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:12.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:12.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.317+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/31725526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 msgr2=0x7f6f040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.317+0000 7f6f0aa4d700 1 --2- 172.21.15.190:0/31725526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 0x7f6f040f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6ef4009fb0 tx=0x7f6ef4009b30).stop 2024-01-16T17:13:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.317+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/31725526 shutdown_connections 2024-01-16T17:13:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.317+0000 7f6f0aa4d700 1 --2- 172.21.15.190:0/31725526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 0x7f6f040f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.317+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/31725526 >> 172.21.15.190:0/31725526 conn(0x7f6f040f0b70 msgr2=0x7f6f040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.317+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/31725526 shutdown_connections 2024-01-16T17:13:13.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.317+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/31725526 wait complete. 2024-01-16T17:13:13.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.318+0000 7f6f0aa4d700 1 Processor -- start 2024-01-16T17:13:13.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.318+0000 7f6f0aa4d700 1 -- start start 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.318+0000 7f6f0aa4d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 0x7f6f04105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.318+0000 7f6f0aa4d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04105b20 con 0x7f6f040f5410 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f03fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 0x7f6f04105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f03fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 0x7f6f04105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60120/0 (socket says 172.21.15.190:60120) 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f03fff700 1 -- 172.21.15.190:0/2531092442 learned_addr learned my addr 172.21.15.190:0/2531092442 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f03fff700 1 -- 172.21.15.190:0/2531092442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef4005040 con 0x7f6f040f5410 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f03fff700 1 --2- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 0x7f6f04105630 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f6ef4013510 tx=0x7f6ef400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f00ff9700 1 -- 172.21.15.190:0/2531092442 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ef4009320 con 0x7f6f040f5410 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f04105d20 con 0x7f6f040f5410 2024-01-16T17:13:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f00ff9700 1 -- 172.21.15.190:0/2531092442 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ef4013f00 con 0x7f6f040f5410 2024-01-16T17:13:13.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f00ff9700 1 -- 172.21.15.190:0/2531092442 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ef4014d90 con 0x7f6f040f5410 2024-01-16T17:13:13.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.319+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f041061b0 con 0x7f6f040f5410 2024-01-16T17:13:13.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.320+0000 7f6f00ff9700 1 -- 172.21.15.190:0/2531092442 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ef40095f0 con 0x7f6f040f5410 2024-01-16T17:13:13.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.320+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f04045ad0 con 0x7f6f040f5410 2024-01-16T17:13:13.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.320+0000 7f6f00ff9700 1 --2- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6eec034370 0x7f6eec036830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:13.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.321+0000 7f6f00ff9700 1 -- 172.21.15.190:0/2531092442 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ef4040fc0 con 0x7f6f040f5410 2024-01-16T17:13:13.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.321+0000 7f6f037fe700 1 --2- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6eec034370 0x7f6eec036830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:13.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.321+0000 7f6f037fe700 1 --2- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6eec034370 0x7f6eec036830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f6ef80096a0 tx=0x7f6ef8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:13.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.325+0000 7f6f00ff9700 1 -- 172.21.15.190:0/2531092442 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ef4014420 con 0x7f6f040f5410 2024-01-16T17:13:13.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.537+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f040fae60 con 0x7f6f040f5410 2024-01-16T17:13:13.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.537+0000 7f6f00ff9700 1 -- 172.21.15.190:0/2531092442 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6ef4010230 con 0x7f6f040f5410 2024-01-16T17:13:13.518 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:13.518 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:13.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6eec034370 msgr2=0x7f6eec036830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:13.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 --2- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6eec034370 0x7f6eec036830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f6ef80096a0 tx=0x7f6ef8006b40).stop 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 msgr2=0x7f6f04105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 --2- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 0x7f6f04105630 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f6ef4013510 tx=0x7f6ef400b040).stop 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 shutdown_connections 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 --2- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6eec034370 0x7f6eec036830 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 --2- 172.21.15.190:0/2531092442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6f040f5410 0x7f6f04105630 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 >> 172.21.15.190:0/2531092442 conn(0x7f6f040f0b70 msgr2=0x7f6f040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 shutdown_connections 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:13.539+0000 7f6f0aa4d700 1 -- 172.21.15.190:0/2531092442 wait complete. 2024-01-16T17:13:13.520 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:13.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:11.804330+0000 mgr.smithi154.sllivz (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:14.945 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:14.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:14.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:13.538349+0000 mon.smithi154 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.190:0/2531092442' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:15.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:13.804704+0000 mgr.smithi154.sllivz (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:16.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.096+0000 7f7c0c687700 1 -- 172.21.15.190:0/56779121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 msgr2=0x7f7c040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:16.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.096+0000 7f7c0c687700 1 --2- 172.21.15.190:0/56779121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 0x7f7c040f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f7c00009fb0 tx=0x7f7c00009b30).stop 2024-01-16T17:13:16.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.096+0000 7f7c0c687700 1 -- 172.21.15.190:0/56779121 shutdown_connections 2024-01-16T17:13:16.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.096+0000 7f7c0c687700 1 --2- 172.21.15.190:0/56779121 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 0x7f7c040f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:16.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.096+0000 7f7c0c687700 1 -- 172.21.15.190:0/56779121 >> 172.21.15.190:0/56779121 conn(0x7f7c040f1c70 msgr2=0x7f7c040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:16.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.096+0000 7f7c0c687700 1 -- 172.21.15.190:0/56779121 shutdown_connections 2024-01-16T17:13:16.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.096+0000 7f7c0c687700 1 -- 172.21.15.190:0/56779121 wait complete. 2024-01-16T17:13:16.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0c687700 1 Processor -- start 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0c687700 1 -- start start 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0c687700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 0x7f7c041066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0c687700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c04106bc0 con 0x7f7c040f6510 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0a423700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 0x7f7c041066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0a423700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 0x7f7c041066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60146/0 (socket says 172.21.15.190:60146) 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0a423700 1 -- 172.21.15.190:0/2564710927 learned_addr learned my addr 172.21.15.190:0/2564710927 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0a423700 1 -- 172.21.15.190:0/2564710927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00005040 con 0x7f7c040f6510 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.097+0000 7f7c0a423700 1 --2- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 0x7f7c041066d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7c00014000 tx=0x7f7c0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.098+0000 7f7bfaffd700 1 -- 172.21.15.190:0/2564710927 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c000095f0 con 0x7f7c040f6510 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.098+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c04106dc0 con 0x7f7c040f6510 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.098+0000 7f7bfaffd700 1 -- 172.21.15.190:0/2564710927 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c00007e70 con 0x7f7c040f6510 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.098+0000 7f7bfaffd700 1 -- 172.21.15.190:0/2564710927 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c00014ea0 con 0x7f7c040f6510 2024-01-16T17:13:16.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.098+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c04107250 con 0x7f7c040f6510 2024-01-16T17:13:16.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.098+0000 7f7bfaffd700 1 -- 172.21.15.190:0/2564710927 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c00025070 con 0x7f7c040f6510 2024-01-16T17:13:16.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.098+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c04045ad0 con 0x7f7c040f6510 2024-01-16T17:13:16.082 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.099+0000 7f7bfaffd700 1 --2- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7bf002ff60 0x7f7bf0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:16.082 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.099+0000 7f7bfaffd700 1 -- 172.21.15.190:0/2564710927 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c00040940 con 0x7f7c040f6510 2024-01-16T17:13:16.082 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.099+0000 7f7c09c22700 1 --2- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7bf002ff60 0x7f7bf0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:16.082 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.099+0000 7f7c09c22700 1 --2- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7bf002ff60 0x7f7bf0032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7bf40096a0 tx=0x7f7bf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:16.085 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.103+0000 7f7bfaffd700 1 -- 172.21.15.190:0/2564710927 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c000079c0 con 0x7f7c040f6510 2024-01-16T17:13:16.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.309+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c041080a0 con 0x7f7c040f6510 2024-01-16T17:13:16.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.309+0000 7f7bfaffd700 1 -- 172.21.15.190:0/2564710927 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7c0000f300 con 0x7f7c040f6510 2024-01-16T17:13:16.291 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:16.291 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:16.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7bf002ff60 msgr2=0x7f7bf0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:16.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 --2- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7bf002ff60 0x7f7bf0032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7bf40096a0 tx=0x7f7bf4006b40).stop 2024-01-16T17:13:16.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 msgr2=0x7f7c041066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:16.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 --2- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 0x7f7c041066d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7c00014000 tx=0x7f7c0000b040).stop 2024-01-16T17:13:16.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 shutdown_connections 2024-01-16T17:13:16.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 --2- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7bf002ff60 0x7f7bf0032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:16.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 --2- 172.21.15.190:0/2564710927 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c040f6510 0x7f7c041066d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:16.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 >> 172.21.15.190:0/2564710927 conn(0x7f7c040f1c70 msgr2=0x7f7c040f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:16.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 shutdown_connections 2024-01-16T17:13:16.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:16.311+0000 7f7c0c687700 1 -- 172.21.15.190:0/2564710927 wait complete. 2024-01-16T17:13:16.293 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:16.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:16 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:16.311320+0000 mon.smithi154 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.190:0/2564710927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:17.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:17.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:17.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:15.805062+0000 mgr.smithi154.sllivz (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.857+0000 7fb82a573700 1 -- 172.21.15.190:0/3956695014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 msgr2=0x7fb8240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.857+0000 7fb82a573700 1 --2- 172.21.15.190:0/3956695014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 0x7fb8240f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb814009fb0 tx=0x7fb814009b30).stop 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.857+0000 7fb82a573700 1 -- 172.21.15.190:0/3956695014 shutdown_connections 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.857+0000 7fb82a573700 1 --2- 172.21.15.190:0/3956695014 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 0x7fb8240f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.857+0000 7fb82a573700 1 -- 172.21.15.190:0/3956695014 >> 172.21.15.190:0/3956695014 conn(0x7fb8240f1c70 msgr2=0x7fb8240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.857+0000 7fb82a573700 1 -- 172.21.15.190:0/3956695014 shutdown_connections 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.857+0000 7fb82a573700 1 -- 172.21.15.190:0/3956695014 wait complete. 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.858+0000 7fb82a573700 1 Processor -- start 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.858+0000 7fb82a573700 1 -- start start 2024-01-16T17:13:18.841 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.858+0000 7fb82a573700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 0x7fb824106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.858+0000 7fb82a573700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb824106c20 con 0x7fb8240f6510 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb823fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 0x7fb824106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb823fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 0x7fb824106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60156/0 (socket says 172.21.15.190:60156) 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb823fff700 1 -- 172.21.15.190:0/794012459 learned_addr learned my addr 172.21.15.190:0/794012459 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb823fff700 1 -- 172.21.15.190:0/794012459 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb814005040 con 0x7fb8240f6510 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb823fff700 1 --2- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 0x7fb824106730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb814014000 tx=0x7fb81400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb820ff9700 1 -- 172.21.15.190:0/794012459 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8140095f0 con 0x7fb8240f6510 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb820ff9700 1 -- 172.21.15.190:0/794012459 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb814007e70 con 0x7fb8240f6510 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb824106e20 con 0x7fb8240f6510 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb820ff9700 1 -- 172.21.15.190:0/794012459 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb814014ea0 con 0x7fb8240f6510 2024-01-16T17:13:18.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.859+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8241072b0 con 0x7fb8240f6510 2024-01-16T17:13:18.843 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.860+0000 7fb820ff9700 1 -- 172.21.15.190:0/794012459 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb814025070 con 0x7fb8240f6510 2024-01-16T17:13:18.843 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.860+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb824045ad0 con 0x7fb8240f6510 2024-01-16T17:13:18.843 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.860+0000 7fb820ff9700 1 --2- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb80c02ff60 0x7fb80c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:18.843 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.860+0000 7fb820ff9700 1 -- 172.21.15.190:0/794012459 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8140409f0 con 0x7fb8240f6510 2024-01-16T17:13:18.843 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.861+0000 7fb8237fe700 1 --2- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb80c02ff60 0x7fb80c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:18.843 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.861+0000 7fb8237fe700 1 --2- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb80c02ff60 0x7fb80c032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb8180096a0 tx=0x7fb818006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:18.846 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:18.865+0000 7fb820ff9700 1 -- 172.21.15.190:0/794012459 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb81400f4e0 con 0x7fb8240f6510 2024-01-16T17:13:19.051 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.070+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8240fbf60 con 0x7fb8240f6510 2024-01-16T17:13:19.051 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.070+0000 7fb820ff9700 1 -- 172.21.15.190:0/794012459 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb81400f300 con 0x7fb8240f6510 2024-01-16T17:13:19.052 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:19.052 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:19.053 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb80c02ff60 msgr2=0x7fb80c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 --2- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb80c02ff60 0x7fb80c032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb8180096a0 tx=0x7fb818006b40).stop 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 msgr2=0x7fb824106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 --2- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 0x7fb824106730 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb814014000 tx=0x7fb81400b040).stop 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 shutdown_connections 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 --2- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb80c02ff60 0x7fb80c032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 --2- 172.21.15.190:0/794012459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8240f6510 0x7fb824106730 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 >> 172.21.15.190:0/794012459 conn(0x7fb8240f1c70 msgr2=0x7fb8240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 shutdown_connections 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:19.072+0000 7fb82a573700 1 -- 172.21.15.190:0/794012459 wait complete. 2024-01-16T17:13:19.054 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:19.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:17.805354+0000 mgr.smithi154.sllivz (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:19.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:19.072511+0000 mon.smithi154 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.190:0/794012459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:20.442 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:20.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 -- 172.21.15.190:0/273648242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 msgr2=0x7f02e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 --2- 172.21.15.190:0/273648242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 0x7f02e80f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f02dc009fb0 tx=0x7f02dc009b30).stop 2024-01-16T17:13:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 -- 172.21.15.190:0/273648242 shutdown_connections 2024-01-16T17:13:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 --2- 172.21.15.190:0/273648242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 0x7f02e80f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 -- 172.21.15.190:0/273648242 >> 172.21.15.190:0/273648242 conn(0x7f02e80f0b70 msgr2=0x7f02e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 -- 172.21.15.190:0/273648242 shutdown_connections 2024-01-16T17:13:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 -- 172.21.15.190:0/273648242 wait complete. 2024-01-16T17:13:21.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 Processor -- start 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.553+0000 7f02ef228700 1 -- start start 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02ef228700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 0x7f02e8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02ef228700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8106c10 con 0x7f02e80f5410 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02ecfc4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 0x7f02e8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02ecfc4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 0x7f02e8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60170/0 (socket says 172.21.15.190:60170) 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02ecfc4700 1 -- 172.21.15.190:0/1257060402 learned_addr learned my addr 172.21.15.190:0/1257060402 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02ecfc4700 1 -- 172.21.15.190:0/1257060402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc005040 con 0x7f02e80f5410 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02ecfc4700 1 --2- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 0x7f02e8106720 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f02dc013510 tx=0x7f02dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02e57fa700 1 -- 172.21.15.190:0/1257060402 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02dc009320 con 0x7f02e80f5410 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e8106e10 con 0x7f02e80f5410 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02e57fa700 1 -- 172.21.15.190:0/1257060402 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02dc013f00 con 0x7f02e80f5410 2024-01-16T17:13:21.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.554+0000 7f02e57fa700 1 -- 172.21.15.190:0/1257060402 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02dc014d90 con 0x7f02e80f5410 2024-01-16T17:13:21.538 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.555+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e81072a0 con 0x7f02e80f5410 2024-01-16T17:13:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.555+0000 7f02e57fa700 1 -- 172.21.15.190:0/1257060402 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02dc0263a0 con 0x7f02e80f5410 2024-01-16T17:13:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.555+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e8045ad0 con 0x7f02e80f5410 2024-01-16T17:13:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.556+0000 7f02e57fa700 1 --2- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.556+0000 7f02e57fa700 1 -- 172.21.15.190:0/1257060402 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02dc0413c0 con 0x7f02e80f5410 2024-01-16T17:13:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.556+0000 7f02e7fff700 1 --2- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:21.539 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.556+0000 7f02e7fff700 1 --2- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f02d802ff60 0x7f02d8032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f02d40096a0 tx=0x7f02d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:21.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.560+0000 7f02e57fa700 1 -- 172.21.15.190:0/1257060402 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02dc0126f0 con 0x7f02e80f5410 2024-01-16T17:13:21.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.773+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02e8108150 con 0x7f02e80f5410 2024-01-16T17:13:21.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.773+0000 7f02e57fa700 1 -- 172.21.15.190:0/1257060402 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f02dc012330 con 0x7f02e80f5410 2024-01-16T17:13:21.755 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:21.755 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:21.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f02d802ff60 msgr2=0x7f02d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 --2- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f02d802ff60 0x7f02d8032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f02d40096a0 tx=0x7f02d4006b40).stop 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 msgr2=0x7f02e8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 --2- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 0x7f02e8106720 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f02dc013510 tx=0x7f02dc00b040).stop 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 shutdown_connections 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 --2- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 --2- 172.21.15.190:0/1257060402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02e80f5410 0x7f02e8106720 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 >> 172.21.15.190:0/1257060402 conn(0x7f02e80f0b70 msgr2=0x7f02e805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 shutdown_connections 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:21.775+0000 7f02ef228700 1 -- 172.21.15.190:0/1257060402 wait complete. 2024-01-16T17:13:21.757 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:21.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:19.805626+0000 mgr.smithi154.sllivz (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:22.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:22 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:21.775307+0000 mon.smithi154 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.190:0/1257060402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:23.125 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:23.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:23.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:21.805991+0000 mgr.smithi154.sllivz (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:24.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.210+0000 7f7321155700 1 -- 172.21.15.190:0/2653640507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 msgr2=0x7f731c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:24.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.210+0000 7f7321155700 1 --2- 172.21.15.190:0/2653640507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 0x7f731c0f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f730c009fb0 tx=0x7f730c009b30).stop 2024-01-16T17:13:24.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.210+0000 7f7321155700 1 -- 172.21.15.190:0/2653640507 shutdown_connections 2024-01-16T17:13:24.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.210+0000 7f7321155700 1 --2- 172.21.15.190:0/2653640507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 0x7f731c0f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:24.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.210+0000 7f7321155700 1 -- 172.21.15.190:0/2653640507 >> 172.21.15.190:0/2653640507 conn(0x7f731c0f0b70 msgr2=0x7f731c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:24.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.210+0000 7f7321155700 1 -- 172.21.15.190:0/2653640507 shutdown_connections 2024-01-16T17:13:24.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.210+0000 7f7321155700 1 -- 172.21.15.190:0/2653640507 wait complete. 2024-01-16T17:13:24.193 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.211+0000 7f7321155700 1 Processor -- start 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.211+0000 7f7321155700 1 -- start start 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.211+0000 7f7321155700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 0x7f731c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.211+0000 7f7321155700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c060220 con 0x7f731c0f5410 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.211+0000 7f731ad9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 0x7f731c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.211+0000 7f731ad9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 0x7f731c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33900/0 (socket says 172.21.15.190:33900) 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.211+0000 7f731ad9d700 1 -- 172.21.15.190:0/4182109501 learned_addr learned my addr 172.21.15.190:0/4182109501 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.211+0000 7f731ad9d700 1 -- 172.21.15.190:0/4182109501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c005040 con 0x7f731c0f5410 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.212+0000 7f731ad9d700 1 --2- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 0x7f731c109450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f730c013510 tx=0x7f730c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.212+0000 7f73037fe700 1 -- 172.21.15.190:0/4182109501 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f730c009320 con 0x7f731c0f5410 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.212+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c060420 con 0x7f731c0f5410 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.212+0000 7f73037fe700 1 -- 172.21.15.190:0/4182109501 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f730c013f00 con 0x7f731c0f5410 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.212+0000 7f73037fe700 1 -- 172.21.15.190:0/4182109501 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f730c014d90 con 0x7f731c0f5410 2024-01-16T17:13:24.194 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.212+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c109b00 con 0x7f731c0f5410 2024-01-16T17:13:24.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.212+0000 7f73037fe700 1 -- 172.21.15.190:0/4182109501 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f730c014420 con 0x7f731c0f5410 2024-01-16T17:13:24.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.213+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c059100 con 0x7f731c0f5410 2024-01-16T17:13:24.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.213+0000 7f73037fe700 1 --2- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f730402ff60 0x7f7304032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:24.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.213+0000 7f73037fe700 1 -- 172.21.15.190:0/4182109501 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f730c0410c0 con 0x7f731c0f5410 2024-01-16T17:13:24.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.213+0000 7f731a59c700 1 --2- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f730402ff60 0x7f7304032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:24.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.213+0000 7f731a59c700 1 --2- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f730402ff60 0x7f7304032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f73100096a0 tx=0x7f7310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:24.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.218+0000 7f73037fe700 1 -- 172.21.15.190:0/4182109501 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f730c0095f0 con 0x7f731c0f5410 2024-01-16T17:13:24.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.426+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f731c045ad0 con 0x7f731c0f5410 2024-01-16T17:13:24.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.426+0000 7f73037fe700 1 -- 172.21.15.190:0/4182109501 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f730c01db70 con 0x7f731c0f5410 2024-01-16T17:13:24.408 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:24.408 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f730402ff60 msgr2=0x7f7304032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 --2- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f730402ff60 0x7f7304032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f73100096a0 tx=0x7f7310006b40).stop 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 msgr2=0x7f731c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 --2- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 0x7f731c109450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f730c013510 tx=0x7f730c00b040).stop 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 shutdown_connections 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 --2- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f730402ff60 0x7f7304032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 --2- 172.21.15.190:0/4182109501 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f731c0f5410 0x7f731c109450 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 >> 172.21.15.190:0/4182109501 conn(0x7f731c0f0b70 msgr2=0x7f731c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 shutdown_connections 2024-01-16T17:13:24.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:24.428+0000 7f7321155700 1 -- 172.21.15.190:0/4182109501 wait complete. 2024-01-16T17:13:24.411 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:24.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:24 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:24.428380+0000 mon.smithi154 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.190:0/4182109501' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:25.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:25.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:25.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:23.806385+0000 mgr.smithi154.sllivz (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:26.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.989+0000 7f0e44133700 1 -- 172.21.15.190:0/1639896759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 msgr2=0x7f0e3c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:26.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.989+0000 7f0e44133700 1 --2- 172.21.15.190:0/1639896759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 0x7f0e3c0f90f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f0e2c009fb0 tx=0x7f0e2c009b30).stop 2024-01-16T17:13:26.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.989+0000 7f0e44133700 1 -- 172.21.15.190:0/1639896759 shutdown_connections 2024-01-16T17:13:26.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.989+0000 7f0e44133700 1 --2- 172.21.15.190:0/1639896759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 0x7f0e3c0f90f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:26.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.989+0000 7f0e44133700 1 -- 172.21.15.190:0/1639896759 >> 172.21.15.190:0/1639896759 conn(0x7f0e3c0f0b70 msgr2=0x7f0e3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:26.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.989+0000 7f0e44133700 1 -- 172.21.15.190:0/1639896759 shutdown_connections 2024-01-16T17:13:26.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.989+0000 7f0e44133700 1 -- 172.21.15.190:0/1639896759 wait complete. 2024-01-16T17:13:26.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.990+0000 7f0e44133700 1 Processor -- start 2024-01-16T17:13:26.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.990+0000 7f0e44133700 1 -- start start 2024-01-16T17:13:26.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.990+0000 7f0e44133700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 0x7f0e3c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.990+0000 7f0e44133700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c106bd0 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.990+0000 7f0e41ecf700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 0x7f0e3c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.990+0000 7f0e41ecf700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 0x7f0e3c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33918/0 (socket says 172.21.15.190:33918) 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.990+0000 7f0e41ecf700 1 -- 172.21.15.190:0/943980376 learned_addr learned my addr 172.21.15.190:0/943980376 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.990+0000 7f0e41ecf700 1 -- 172.21.15.190:0/943980376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c005040 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.991+0000 7f0e41ecf700 1 --2- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 0x7f0e3c1066e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f0e2c013510 tx=0x7f0e2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.991+0000 7f0e327fc700 1 -- 172.21.15.190:0/943980376 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e2c009320 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.991+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c106dd0 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.991+0000 7f0e327fc700 1 -- 172.21.15.190:0/943980376 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e2c013f00 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.991+0000 7f0e327fc700 1 -- 172.21.15.190:0/943980376 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e2c014dc0 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.991+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c107260 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.992+0000 7f0e327fc700 1 -- 172.21.15.190:0/943980376 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e2c0095f0 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.992+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c045ad0 con 0x7f0e3c060280 2024-01-16T17:13:26.973 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.992+0000 7f0e327fc700 1 --2- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e2802ff60 0x7f0e28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:26.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.992+0000 7f0e327fc700 1 -- 172.21.15.190:0/943980376 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e2c041000 con 0x7f0e3c060280 2024-01-16T17:13:26.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.992+0000 7f0e416ce700 1 --2- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e2802ff60 0x7f0e28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:26.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.992+0000 7f0e416ce700 1 --2- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e2802ff60 0x7f0e28032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f0e380096a0 tx=0x7f0e38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:26.976 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:26.996+0000 7f0e327fc700 1 -- 172.21.15.190:0/943980376 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e2c01c790 con 0x7f0e3c060280 2024-01-16T17:13:27.182 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.202+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e3c108110 con 0x7f0e3c060280 2024-01-16T17:13:27.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.203+0000 7f0e327fc700 1 -- 172.21.15.190:0/943980376 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0e2c010230 con 0x7f0e3c060280 2024-01-16T17:13:27.183 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:27.183 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e2802ff60 msgr2=0x7f0e28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 --2- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e2802ff60 0x7f0e28032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f0e380096a0 tx=0x7f0e38006b40).stop 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 msgr2=0x7f0e3c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 --2- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 0x7f0e3c1066e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f0e2c013510 tx=0x7f0e2c00b040).stop 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 shutdown_connections 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 --2- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e2802ff60 0x7f0e28032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 --2- 172.21.15.190:0/943980376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e3c060280 0x7f0e3c1066e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 >> 172.21.15.190:0/943980376 conn(0x7f0e3c0f0b70 msgr2=0x7f0e3c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 shutdown_connections 2024-01-16T17:13:27.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:27.204+0000 7f0e44133700 1 -- 172.21.15.190:0/943980376 wait complete. 2024-01-16T17:13:27.186 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:27.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:25.806636+0000 mgr.smithi154.sllivz (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:27.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:27.203848+0000 mon.smithi154 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.190:0/943980376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:28.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:28.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:29.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.755+0000 7f81b0a6b700 1 -- 172.21.15.190:0/2336287156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 msgr2=0x7f81ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:29.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.755+0000 7f81b0a6b700 1 --2- 172.21.15.190:0/2336287156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 0x7f81ac0f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f8194009fb0 tx=0x7f8194009b30).stop 2024-01-16T17:13:29.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.755+0000 7f81b0a6b700 1 -- 172.21.15.190:0/2336287156 shutdown_connections 2024-01-16T17:13:29.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.755+0000 7f81b0a6b700 1 --2- 172.21.15.190:0/2336287156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 0x7f81ac0f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.755+0000 7f81b0a6b700 1 -- 172.21.15.190:0/2336287156 >> 172.21.15.190:0/2336287156 conn(0x7f81ac0f0b70 msgr2=0x7f81ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.755+0000 7f81b0a6b700 1 -- 172.21.15.190:0/2336287156 shutdown_connections 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.755+0000 7f81b0a6b700 1 -- 172.21.15.190:0/2336287156 wait complete. 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.756+0000 7f81b0a6b700 1 Processor -- start 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.756+0000 7f81b0a6b700 1 -- start start 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.756+0000 7f81b0a6b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 0x7f81ac109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.756+0000 7f81b0a6b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac060220 con 0x7f81ac0f5410 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.756+0000 7f81aa59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 0x7f81ac109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.756+0000 7f81aa59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 0x7f81ac109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33924/0 (socket says 172.21.15.190:33924) 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.756+0000 7f81aa59c700 1 -- 172.21.15.190:0/4257776950 learned_addr learned my addr 172.21.15.190:0/4257776950 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.756+0000 7f81aa59c700 1 -- 172.21.15.190:0/4257776950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8194005040 con 0x7f81ac0f5410 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.757+0000 7f81aa59c700 1 --2- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 0x7f81ac109450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f8194013510 tx=0x7f819400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.757+0000 7f81a2ffd700 1 -- 172.21.15.190:0/4257776950 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8194009320 con 0x7f81ac0f5410 2024-01-16T17:13:29.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.757+0000 7f81a2ffd700 1 -- 172.21.15.190:0/4257776950 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8194013f00 con 0x7f81ac0f5410 2024-01-16T17:13:29.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.757+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ac060420 con 0x7f81ac0f5410 2024-01-16T17:13:29.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.757+0000 7f81a2ffd700 1 -- 172.21.15.190:0/4257776950 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8194014d90 con 0x7f81ac0f5410 2024-01-16T17:13:29.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.757+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac109b00 con 0x7f81ac0f5410 2024-01-16T17:13:29.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.758+0000 7f81a2ffd700 1 -- 172.21.15.190:0/4257776950 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81940095f0 con 0x7f81ac0f5410 2024-01-16T17:13:29.740 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.758+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ac059100 con 0x7f81ac0f5410 2024-01-16T17:13:29.740 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.758+0000 7f81a2ffd700 1 --2- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f819802ff60 0x7f8198032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:29.740 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.758+0000 7f81a2ffd700 1 -- 172.21.15.190:0/4257776950 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81940410c0 con 0x7f81ac0f5410 2024-01-16T17:13:29.740 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.758+0000 7f81a9d9b700 1 --2- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f819802ff60 0x7f8198032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:29.740 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.758+0000 7f81a9d9b700 1 --2- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f819802ff60 0x7f8198032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f819c0096a0 tx=0x7f819c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:29.743 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.763+0000 7f81a2ffd700 1 -- 172.21.15.190:0/4257776950 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f819401d7f0 con 0x7f81ac0f5410 2024-01-16T17:13:29.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.973+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81ac045ad0 con 0x7f81ac0f5410 2024-01-16T17:13:29.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.973+0000 7f81a2ffd700 1 -- 172.21.15.190:0/4257776950 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8194010230 con 0x7f81ac0f5410 2024-01-16T17:13:29.954 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:29.954 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:29.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f819802ff60 msgr2=0x7f8198032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:29.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 --2- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f819802ff60 0x7f8198032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f819c0096a0 tx=0x7f819c006b40).stop 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 msgr2=0x7f81ac109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 --2- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 0x7f81ac109450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f8194013510 tx=0x7f819400b040).stop 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 shutdown_connections 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 --2- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f819802ff60 0x7f8198032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 --2- 172.21.15.190:0/4257776950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81ac0f5410 0x7f81ac109450 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 >> 172.21.15.190:0/4257776950 conn(0x7f81ac0f0b70 msgr2=0x7f81ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 shutdown_connections 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:29.975+0000 7f81b0a6b700 1 -- 172.21.15.190:0/4257776950 wait complete. 2024-01-16T17:13:29.956 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:29.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:27.806872+0000 mgr.smithi154.sllivz (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:30.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:30 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:29.974479+0000 mon.smithi154 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.190:0/4257776950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:31.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:31.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:31.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:29.807212+0000 mgr.smithi154.sllivz (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:32.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.422+0000 7fe261a0c700 1 -- 172.21.15.190:0/1377719300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 msgr2=0x7fe25c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:32.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.422+0000 7fe261a0c700 1 --2- 172.21.15.190:0/1377719300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 0x7fe25c0f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fe250009fb0 tx=0x7fe250009b30).stop 2024-01-16T17:13:32.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.422+0000 7fe261a0c700 1 -- 172.21.15.190:0/1377719300 shutdown_connections 2024-01-16T17:13:32.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.422+0000 7fe261a0c700 1 --2- 172.21.15.190:0/1377719300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 0x7fe25c0f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:32.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.422+0000 7fe261a0c700 1 -- 172.21.15.190:0/1377719300 >> 172.21.15.190:0/1377719300 conn(0x7fe25c0f0b70 msgr2=0x7fe25c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:32.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.422+0000 7fe261a0c700 1 -- 172.21.15.190:0/1377719300 shutdown_connections 2024-01-16T17:13:32.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.422+0000 7fe261a0c700 1 -- 172.21.15.190:0/1377719300 wait complete. 2024-01-16T17:13:32.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.423+0000 7fe261a0c700 1 Processor -- start 2024-01-16T17:13:32.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.423+0000 7fe261a0c700 1 -- start start 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.423+0000 7fe261a0c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 0x7fe25c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.423+0000 7fe261a0c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c105b20 con 0x7fe25c0f5410 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.423+0000 7fe25affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 0x7fe25c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.423+0000 7fe25affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 0x7fe25c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:49164/0 (socket says 172.21.15.190:49164) 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.423+0000 7fe25affd700 1 -- 172.21.15.190:0/79851414 learned_addr learned my addr 172.21.15.190:0/79851414 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.423+0000 7fe25affd700 1 -- 172.21.15.190:0/79851414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe250005040 con 0x7fe25c0f5410 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.424+0000 7fe25affd700 1 --2- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 0x7fe25c105630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe250013510 tx=0x7fe25000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.424+0000 7fe2437fe700 1 -- 172.21.15.190:0/79851414 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe250009320 con 0x7fe25c0f5410 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.424+0000 7fe2437fe700 1 -- 172.21.15.190:0/79851414 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe250013f00 con 0x7fe25c0f5410 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.424+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c105d20 con 0x7fe25c0f5410 2024-01-16T17:13:32.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.424+0000 7fe2437fe700 1 -- 172.21.15.190:0/79851414 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe250014d90 con 0x7fe25c0f5410 2024-01-16T17:13:32.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.424+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c1061b0 con 0x7fe25c0f5410 2024-01-16T17:13:32.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.425+0000 7fe2437fe700 1 -- 172.21.15.190:0/79851414 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2500095f0 con 0x7fe25c0f5410 2024-01-16T17:13:32.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.425+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25c045ad0 con 0x7fe25c0f5410 2024-01-16T17:13:32.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.425+0000 7fe2437fe700 1 --2- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe24402ff60 0x7fe244032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:32.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.425+0000 7fe2437fe700 1 -- 172.21.15.190:0/79851414 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe250041000 con 0x7fe25c0f5410 2024-01-16T17:13:32.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.425+0000 7fe25a7fc700 1 --2- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe24402ff60 0x7fe244032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:32.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.425+0000 7fe25a7fc700 1 --2- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe24402ff60 0x7fe244032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fe24c0096a0 tx=0x7fe24c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:32.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.430+0000 7fe2437fe700 1 -- 172.21.15.190:0/79851414 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe250014420 con 0x7fe25c0f5410 2024-01-16T17:13:32.618 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.638+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe25c0fae60 con 0x7fe25c0f5410 2024-01-16T17:13:32.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.638+0000 7fe2437fe700 1 -- 172.21.15.190:0/79851414 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe25001d7f0 con 0x7fe25c0f5410 2024-01-16T17:13:32.619 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:32.619 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:32.620 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe24402ff60 msgr2=0x7fe244032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:32.620 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 --2- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe24402ff60 0x7fe244032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fe24c0096a0 tx=0x7fe24c006b40).stop 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 msgr2=0x7fe25c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 --2- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 0x7fe25c105630 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fe250013510 tx=0x7fe25000b040).stop 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 shutdown_connections 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 --2- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe24402ff60 0x7fe244032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 --2- 172.21.15.190:0/79851414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe25c0f5410 0x7fe25c105630 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 >> 172.21.15.190:0/79851414 conn(0x7fe25c0f0b70 msgr2=0x7fe25c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 shutdown_connections 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:32.640+0000 7fe261a0c700 1 -- 172.21.15.190:0/79851414 wait complete. 2024-01-16T17:13:32.621 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:33.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:31.807583+0000 mgr.smithi154.sllivz (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:33.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:32.639444+0000 mon.smithi154 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.190:0/79851414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:34.002 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:34.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:35.177 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 -- 172.21.15.190:0/2491982823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 msgr2=0x7fddf00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:35.177 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 --2- 172.21.15.190:0/2491982823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 0x7fddf00f55c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fddec009fb0 tx=0x7fddec009b30).stop 2024-01-16T17:13:35.177 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 -- 172.21.15.190:0/2491982823 shutdown_connections 2024-01-16T17:13:35.177 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 --2- 172.21.15.190:0/2491982823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 0x7fddf00f55c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:35.177 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 -- 172.21.15.190:0/2491982823 >> 172.21.15.190:0/2491982823 conn(0x7fddf00f09a0 msgr2=0x7fddf00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 -- 172.21.15.190:0/2491982823 shutdown_connections 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 -- 172.21.15.190:0/2491982823 wait complete. 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 Processor -- start 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 -- start start 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 0x7fddf01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.195+0000 7fddf74d6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0060220 con 0x7fddf00f51a0 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fddf64d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 0x7fddf01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fddf64d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 0x7fddf01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:49184/0 (socket says 172.21.15.190:49184) 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fddf64d4700 1 -- 172.21.15.190:0/3410560955 learned_addr learned my addr 172.21.15.190:0/3410560955 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fddf64d4700 1 -- 172.21.15.190:0/3410560955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec005040 con 0x7fddf00f51a0 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fddf64d4700 1 --2- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 0x7fddf01091d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fddec013510 tx=0x7fddec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fdde6ffd700 1 -- 172.21.15.190:0/3410560955 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddec009320 con 0x7fddf00f51a0 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf0060420 con 0x7fddf00f51a0 2024-01-16T17:13:35.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fdde6ffd700 1 -- 172.21.15.190:0/3410560955 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fddec013f00 con 0x7fddf00f51a0 2024-01-16T17:13:35.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fdde6ffd700 1 -- 172.21.15.190:0/3410560955 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddec014dc0 con 0x7fddf00f51a0 2024-01-16T17:13:35.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.196+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf0109880 con 0x7fddf00f51a0 2024-01-16T17:13:35.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.197+0000 7fdde6ffd700 1 -- 172.21.15.190:0/3410560955 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddec0095f0 con 0x7fddf00f51a0 2024-01-16T17:13:35.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.197+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf0059100 con 0x7fddf00f51a0 2024-01-16T17:13:35.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.197+0000 7fdde6ffd700 1 --2- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdddc02ff60 0x7fdddc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:35.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.197+0000 7fdde6ffd700 1 -- 172.21.15.190:0/3410560955 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddec041000 con 0x7fddf00f51a0 2024-01-16T17:13:35.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.198+0000 7fddf5cd3700 1 --2- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdddc02ff60 0x7fdddc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:35.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.198+0000 7fddf5cd3700 1 --2- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdddc02ff60 0x7fdddc032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fdde00096a0 tx=0x7fdde0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:35.182 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.202+0000 7fdde6ffd700 1 -- 172.21.15.190:0/3410560955 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddec01c790 con 0x7fddf00f51a0 2024-01-16T17:13:35.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.408+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fddf0045ad0 con 0x7fddf00f51a0 2024-01-16T17:13:35.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.409+0000 7fdde6ffd700 1 -- 172.21.15.190:0/3410560955 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fddec010230 con 0x7fddf00f51a0 2024-01-16T17:13:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:35.389 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdddc02ff60 msgr2=0x7fdddc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 --2- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdddc02ff60 0x7fdddc032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fdde00096a0 tx=0x7fdde0006b40).stop 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 msgr2=0x7fddf01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 --2- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 0x7fddf01091d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fddec013510 tx=0x7fddec00b040).stop 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 shutdown_connections 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 --2- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdddc02ff60 0x7fdddc032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 --2- 172.21.15.190:0/3410560955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fddf00f51a0 0x7fddf01091d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 >> 172.21.15.190:0/3410560955 conn(0x7fddf00f09a0 msgr2=0x7fddf00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:35.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 shutdown_connections 2024-01-16T17:13:35.392 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:35.410+0000 7fddf74d6700 1 -- 172.21.15.190:0/3410560955 wait complete. 2024-01-16T17:13:35.392 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:35.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:33.807923+0000 mgr.smithi154.sllivz (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:35.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:35.409451+0000 mon.smithi154 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.190:0/3410560955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:36.828 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:36.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.937+0000 7f97c4ba3700 1 -- 172.21.15.190:0/2100189923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 msgr2=0x7f97c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.937+0000 7f97c4ba3700 1 --2- 172.21.15.190:0/2100189923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 0x7f97c00f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f97b4009fb0 tx=0x7f97b4009b30).stop 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.937+0000 7f97c4ba3700 1 -- 172.21.15.190:0/2100189923 shutdown_connections 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.937+0000 7f97c4ba3700 1 --2- 172.21.15.190:0/2100189923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 0x7f97c00f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.937+0000 7f97c4ba3700 1 -- 172.21.15.190:0/2100189923 >> 172.21.15.190:0/2100189923 conn(0x7f97c00f0b70 msgr2=0x7f97c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.937+0000 7f97c4ba3700 1 -- 172.21.15.190:0/2100189923 shutdown_connections 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.937+0000 7f97c4ba3700 1 -- 172.21.15.190:0/2100189923 wait complete. 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.937+0000 7f97c4ba3700 1 Processor -- start 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97c4ba3700 1 -- start start 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97c4ba3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 0x7f97c00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97c4ba3700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c00fb820 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97be59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 0x7f97c00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97be59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 0x7f97c00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:49202/0 (socket says 172.21.15.190:49202) 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97be59c700 1 -- 172.21.15.190:0/1992792628 learned_addr learned my addr 172.21.15.190:0/1992792628 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97be59c700 1 -- 172.21.15.190:0/1992792628 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b4005040 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97be59c700 1 --2- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 0x7f97c00fd3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f97b4013510 tx=0x7f97b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97aeffd700 1 -- 172.21.15.190:0/1992792628 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97b4009320 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c00fb9c0 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97aeffd700 1 -- 172.21.15.190:0/1992792628 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f97b4013f00 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97aeffd700 1 -- 172.21.15.190:0/1992792628 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97b4014d90 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.938+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c00fbe50 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.939+0000 7f97aeffd700 1 -- 172.21.15.190:0/1992792628 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f97b4014420 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.939+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c0059100 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.939+0000 7f97aeffd700 1 --2- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f97a802ff60 0x7f97a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.939+0000 7f97aeffd700 1 -- 172.21.15.190:0/1992792628 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f97b4041000 con 0x7f97c00f5410 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.940+0000 7f97bdd9b700 1 --2- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f97a802ff60 0x7f97a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:37.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.940+0000 7f97bdd9b700 1 --2- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f97a802ff60 0x7f97a8032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f97b00096a0 tx=0x7f97b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:37.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:37.944+0000 7f97aeffd700 1 -- 172.21.15.190:0/1992792628 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f97b40095f0 con 0x7f97c00f5410 2024-01-16T17:13:37.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:35.808295+0000 mgr.smithi154.sllivz (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:38.133 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.153+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97c00fae60 con 0x7f97c00f5410 2024-01-16T17:13:38.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.154+0000 7f97aeffd700 1 -- 172.21.15.190:0/1992792628 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f97b401db70 con 0x7f97c00f5410 2024-01-16T17:13:38.134 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:38.134 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.155+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f97a802ff60 msgr2=0x7f97a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.155+0000 7f97c4ba3700 1 --2- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f97a802ff60 0x7f97a8032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f97b00096a0 tx=0x7f97b0006b40).stop 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.155+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 msgr2=0x7f97c00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.155+0000 7f97c4ba3700 1 --2- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 0x7f97c00fd3a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f97b4013510 tx=0x7f97b400b040).stop 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.156+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 shutdown_connections 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.156+0000 7f97c4ba3700 1 --2- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f97a802ff60 0x7f97a8032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.156+0000 7f97c4ba3700 1 --2- 172.21.15.190:0/1992792628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97c00f5410 0x7f97c00fd3a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.156+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 >> 172.21.15.190:0/1992792628 conn(0x7f97c00f0b70 msgr2=0x7f97c005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.156+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 shutdown_connections 2024-01-16T17:13:38.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:38.156+0000 7f97c4ba3700 1 -- 172.21.15.190:0/1992792628 wait complete. 2024-01-16T17:13:38.137 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:38.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:38 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:38.154947+0000 mon.smithi154 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.190:0/1992792628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:39.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:39.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:39.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:37.808590+0000 mgr.smithi154.sllivz (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:40.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.604+0000 7f11c2294700 1 -- 172.21.15.190:0/2206511359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 msgr2=0x7f11bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.604+0000 7f11c2294700 1 --2- 172.21.15.190:0/2206511359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 0x7f11bc0f6930 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f11ac009fb0 tx=0x7f11ac009b30).stop 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.604+0000 7f11c2294700 1 -- 172.21.15.190:0/2206511359 shutdown_connections 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.604+0000 7f11c2294700 1 --2- 172.21.15.190:0/2206511359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 0x7f11bc0f6930 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.604+0000 7f11c2294700 1 -- 172.21.15.190:0/2206511359 >> 172.21.15.190:0/2206511359 conn(0x7f11bc0f1c70 msgr2=0x7f11bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.605+0000 7f11c2294700 1 -- 172.21.15.190:0/2206511359 shutdown_connections 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.605+0000 7f11c2294700 1 -- 172.21.15.190:0/2206511359 wait complete. 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.605+0000 7f11c2294700 1 Processor -- start 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.605+0000 7f11c2294700 1 -- start start 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11c2294700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 0x7f11bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11c2294700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc106c20 con 0x7f11bc0f6510 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11bbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 0x7f11bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11bbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 0x7f11bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:49230/0 (socket says 172.21.15.190:49230) 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11bbfff700 1 -- 172.21.15.190:0/166171574 learned_addr learned my addr 172.21.15.190:0/166171574 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:40.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11bbfff700 1 -- 172.21.15.190:0/166171574 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac005040 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11bbfff700 1 --2- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 0x7f11bc106730 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f11ac013510 tx=0x7f11ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11b8ff9700 1 -- 172.21.15.190:0/166171574 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11ac009320 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc106e20 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11b8ff9700 1 -- 172.21.15.190:0/166171574 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11ac013f00 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11b8ff9700 1 -- 172.21.15.190:0/166171574 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11ac014d90 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.606+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc1072b0 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.607+0000 7f11b8ff9700 1 -- 172.21.15.190:0/166171574 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11ac014420 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.607+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc045ad0 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.607+0000 7f11b8ff9700 1 --2- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f11a402ff60 0x7f11a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.607+0000 7f11b8ff9700 1 -- 172.21.15.190:0/166171574 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11ac041000 con 0x7f11bc0f6510 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.608+0000 7f11bb7fe700 1 --2- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f11a402ff60 0x7f11a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:40.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.608+0000 7f11bb7fe700 1 --2- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f11a402ff60 0x7f11a4032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f11b00096a0 tx=0x7f11b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:40.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.612+0000 7f11b8ff9700 1 -- 172.21.15.190:0/166171574 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11ac0095f0 con 0x7f11bc0f6510 2024-01-16T17:13:40.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.816+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11bc0fbf60 con 0x7f11bc0f6510 2024-01-16T17:13:40.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.817+0000 7f11b8ff9700 1 -- 172.21.15.190:0/166171574 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f11ac01db70 con 0x7f11bc0f6510 2024-01-16T17:13:40.797 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:40.797 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f11a402ff60 msgr2=0x7f11a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 --2- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f11a402ff60 0x7f11a4032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f11b00096a0 tx=0x7f11b0006b40).stop 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 msgr2=0x7f11bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 --2- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 0x7f11bc106730 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f11ac013510 tx=0x7f11ac00b040).stop 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 shutdown_connections 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 --2- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f11a402ff60 0x7f11a4032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 --2- 172.21.15.190:0/166171574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11bc0f6510 0x7f11bc106730 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 >> 172.21.15.190:0/166171574 conn(0x7f11bc0f1c70 msgr2=0x7f11bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.818+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 shutdown_connections 2024-01-16T17:13:40.799 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:40.819+0000 7f11c2294700 1 -- 172.21.15.190:0/166171574 wait complete. 2024-01-16T17:13:40.800 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:41.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:39.808907+0000 mgr.smithi154.sllivz (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:41.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:40.817725+0000 mon.smithi154 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.190:0/166171574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:42.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:42.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:43.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 -- 172.21.15.190:0/647405521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 msgr2=0x7f1b2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 --2- 172.21.15.190:0/647405521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 0x7f1b2c0f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1b20009fb0 tx=0x7f1b20009b30).stop 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 -- 172.21.15.190:0/647405521 shutdown_connections 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 --2- 172.21.15.190:0/647405521 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 0x7f1b2c0f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 -- 172.21.15.190:0/647405521 >> 172.21.15.190:0/647405521 conn(0x7f1b2c0f1c70 msgr2=0x7f1b2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 -- 172.21.15.190:0/647405521 shutdown_connections 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 -- 172.21.15.190:0/647405521 wait complete. 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 Processor -- start 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.242+0000 7f1b3255d700 1 -- start start 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b3255d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 0x7f1b2c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:43.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b3255d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c106c20 con 0x7f1b2c0f6510 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b2bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 0x7f1b2c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b2bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 0x7f1b2c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:41772/0 (socket says 172.21.15.190:41772) 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b2bfff700 1 -- 172.21.15.190:0/1663951312 learned_addr learned my addr 172.21.15.190:0/1663951312 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b2bfff700 1 -- 172.21.15.190:0/1663951312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b20005040 con 0x7f1b2c0f6510 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b2bfff700 1 --2- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 0x7f1b2c106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1b20013510 tx=0x7f1b2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b28ff9700 1 -- 172.21.15.190:0/1663951312 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b20009320 con 0x7f1b2c0f6510 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c106e20 con 0x7f1b2c0f6510 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b28ff9700 1 -- 172.21.15.190:0/1663951312 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b20013f00 con 0x7f1b2c0f6510 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.243+0000 7f1b28ff9700 1 -- 172.21.15.190:0/1663951312 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b20014d90 con 0x7f1b2c0f6510 2024-01-16T17:13:43.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.244+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c1072b0 con 0x7f1b2c0f6510 2024-01-16T17:13:43.226 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.244+0000 7f1b28ff9700 1 -- 172.21.15.190:0/1663951312 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b200095f0 con 0x7f1b2c0f6510 2024-01-16T17:13:43.226 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.244+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c045ad0 con 0x7f1b2c0f6510 2024-01-16T17:13:43.226 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.244+0000 7f1b28ff9700 1 --2- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1b1402ff60 0x7f1b14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:43.227 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.244+0000 7f1b28ff9700 1 -- 172.21.15.190:0/1663951312 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b20040fe0 con 0x7f1b2c0f6510 2024-01-16T17:13:43.227 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.245+0000 7f1b2b7fe700 1 --2- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1b1402ff60 0x7f1b14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:43.227 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.245+0000 7f1b2b7fe700 1 --2- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1b1402ff60 0x7f1b14032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f1b1c0096a0 tx=0x7f1b1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:43.229 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.249+0000 7f1b28ff9700 1 -- 172.21.15.190:0/1663951312 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b2001d7f0 con 0x7f1b2c0f6510 2024-01-16T17:13:43.435 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.455+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b2c0fbf60 con 0x7f1b2c0f6510 2024-01-16T17:13:43.436 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.456+0000 7f1b28ff9700 1 -- 172.21.15.190:0/1663951312 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1b20010230 con 0x7f1b2c0f6510 2024-01-16T17:13:43.436 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:43.436 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1b1402ff60 msgr2=0x7f1b14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 --2- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1b1402ff60 0x7f1b14032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f1b1c0096a0 tx=0x7f1b1c006b40).stop 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 msgr2=0x7f1b2c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 --2- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 0x7f1b2c106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1b20013510 tx=0x7f1b2000b040).stop 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 shutdown_connections 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 --2- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1b1402ff60 0x7f1b14032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 --2- 172.21.15.190:0/1663951312 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b2c0f6510 0x7f1b2c106730 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 >> 172.21.15.190:0/1663951312 conn(0x7f1b2c0f1c70 msgr2=0x7f1b2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:43.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 shutdown_connections 2024-01-16T17:13:43.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:43.458+0000 7f1b3255d700 1 -- 172.21.15.190:0/1663951312 wait complete. 2024-01-16T17:13:43.439 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:43.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:41.809288+0000 mgr.smithi154.sllivz (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:43.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:43.457188+0000 mon.smithi154 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.190:0/1663951312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:44.910 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:44.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:45.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:43.809591+0000 mgr.smithi154.sllivz (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.062+0000 7f71ce77e700 1 -- 172.21.15.190:0/1788274692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 msgr2=0x7f71c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.062+0000 7f71ce77e700 1 --2- 172.21.15.190:0/1788274692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 0x7f71c80f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f71b8009fb0 tx=0x7f71b8009b30).stop 2024-01-16T17:13:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.062+0000 7f71ce77e700 1 -- 172.21.15.190:0/1788274692 shutdown_connections 2024-01-16T17:13:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.062+0000 7f71ce77e700 1 --2- 172.21.15.190:0/1788274692 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 0x7f71c80f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.062+0000 7f71ce77e700 1 -- 172.21.15.190:0/1788274692 >> 172.21.15.190:0/1788274692 conn(0x7f71c80f0b70 msgr2=0x7f71c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.062+0000 7f71ce77e700 1 -- 172.21.15.190:0/1788274692 shutdown_connections 2024-01-16T17:13:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.062+0000 7f71ce77e700 1 -- 172.21.15.190:0/1788274692 wait complete. 2024-01-16T17:13:46.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.063+0000 7f71ce77e700 1 Processor -- start 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.063+0000 7f71ce77e700 1 -- start start 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.063+0000 7f71ce77e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 0x7f71c8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.063+0000 7f71ce77e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8060240 con 0x7f71c80f5410 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71c7fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 0x7f71c8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71c7fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 0x7f71c8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:41792/0 (socket says 172.21.15.190:41792) 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71c7fff700 1 -- 172.21.15.190:0/3195034718 learned_addr learned my addr 172.21.15.190:0/3195034718 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71c7fff700 1 -- 172.21.15.190:0/3195034718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b8005040 con 0x7f71c80f5410 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71c7fff700 1 --2- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 0x7f71c8109430 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f71b8014000 tx=0x7f71b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71c4ff9700 1 -- 172.21.15.190:0/3195034718 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b80095f0 con 0x7f71c80f5410 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c8060440 con 0x7f71c80f5410 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71c4ff9700 1 -- 172.21.15.190:0/3195034718 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71b8007e70 con 0x7f71c80f5410 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71c4ff9700 1 -- 172.21.15.190:0/3195034718 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b8014ea0 con 0x7f71c80f5410 2024-01-16T17:13:46.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.064+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c8109ae0 con 0x7f71c80f5410 2024-01-16T17:13:46.046 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.065+0000 7f71c4ff9700 1 -- 172.21.15.190:0/3195034718 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71b8025070 con 0x7f71c80f5410 2024-01-16T17:13:46.046 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.065+0000 7f71c4ff9700 1 --2- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f71b402ff60 0x7f71b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:46.046 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.065+0000 7f71c4ff9700 1 -- 172.21.15.190:0/3195034718 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71b8040a30 con 0x7f71c80f5410 2024-01-16T17:13:46.047 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.065+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c8045ad0 con 0x7f71c80f5410 2024-01-16T17:13:46.047 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.065+0000 7f71c77fe700 1 --2- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f71b402ff60 0x7f71b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:46.047 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.066+0000 7f71c77fe700 1 --2- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f71b402ff60 0x7f71b4032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f71b00096a0 tx=0x7f71b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:46.050 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.070+0000 7f71c4ff9700 1 -- 172.21.15.190:0/3195034718 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71b8026350 con 0x7f71c80f5410 2024-01-16T17:13:46.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.276+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71c80faf50 con 0x7f71c80f5410 2024-01-16T17:13:46.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.277+0000 7f71c4ff9700 1 -- 172.21.15.190:0/3195034718 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f71b8012330 con 0x7f71c80f5410 2024-01-16T17:13:46.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:46.257 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:46.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f71b402ff60 msgr2=0x7f71b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:46.258 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 --2- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f71b402ff60 0x7f71b4032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f71b00096a0 tx=0x7f71b0006b40).stop 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 msgr2=0x7f71c8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 --2- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 0x7f71c8109430 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f71b8014000 tx=0x7f71b800b040).stop 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 shutdown_connections 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 --2- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f71b402ff60 0x7f71b4032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 --2- 172.21.15.190:0/3195034718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71c80f5410 0x7f71c8109430 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 >> 172.21.15.190:0/3195034718 conn(0x7f71c80f0b70 msgr2=0x7f71c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 shutdown_connections 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:46.278+0000 7f71ce77e700 1 -- 172.21.15.190:0/3195034718 wait complete. 2024-01-16T17:13:46.259 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:46.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:46 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:46.277639+0000 mon.smithi154 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.190:0/3195034718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:47.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:47.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:47.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:45.809902+0000 mgr.smithi154.sllivz (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.770+0000 7f09ef71a700 1 -- 172.21.15.190:0/1016800318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 msgr2=0x7f09e80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.770+0000 7f09ef71a700 1 --2- 172.21.15.190:0/1016800318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 0x7f09e80f4b60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f09dc009fb0 tx=0x7f09dc009b30).stop 2024-01-16T17:13:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.770+0000 7f09ef71a700 1 -- 172.21.15.190:0/1016800318 shutdown_connections 2024-01-16T17:13:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.770+0000 7f09ef71a700 1 --2- 172.21.15.190:0/1016800318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 0x7f09e80f4b60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.770+0000 7f09ef71a700 1 -- 172.21.15.190:0/1016800318 >> 172.21.15.190:0/1016800318 conn(0x7f09e80f0260 msgr2=0x7f09e80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.770+0000 7f09ef71a700 1 -- 172.21.15.190:0/1016800318 shutdown_connections 2024-01-16T17:13:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.770+0000 7f09ef71a700 1 -- 172.21.15.190:0/1016800318 wait complete. 2024-01-16T17:13:48.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.771+0000 7f09ef71a700 1 Processor -- start 2024-01-16T17:13:48.752 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.771+0000 7f09ef71a700 1 -- start start 2024-01-16T17:13:48.752 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.771+0000 7f09ef71a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 0x7f09e81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:48.752 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.771+0000 7f09ef71a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e8106bc0 con 0x7f09e80f4740 2024-01-16T17:13:48.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.771+0000 7f09ed4b6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 0x7f09e81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:48.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.771+0000 7f09ed4b6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 0x7f09e81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:41804/0 (socket says 172.21.15.190:41804) 2024-01-16T17:13:48.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.771+0000 7f09ed4b6700 1 -- 172.21.15.190:0/2294484973 learned_addr learned my addr 172.21.15.190:0/2294484973 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.772+0000 7f09ed4b6700 1 -- 172.21.15.190:0/2294484973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09dc005040 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.772+0000 7f09ed4b6700 1 --2- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 0x7f09e81066d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f09dc013510 tx=0x7f09dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.772+0000 7f09d9ffb700 1 -- 172.21.15.190:0/2294484973 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09dc009320 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.772+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e8106dc0 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.772+0000 7f09d9ffb700 1 -- 172.21.15.190:0/2294484973 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f09dc013f00 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.772+0000 7f09d9ffb700 1 -- 172.21.15.190:0/2294484973 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09dc014d90 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.772+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e8107250 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.773+0000 7f09d9ffb700 1 -- 172.21.15.190:0/2294484973 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09dc0095f0 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.773+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e8045ad0 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.773+0000 7f09d9ffb700 1 --2- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f09d402ff60 0x7f09d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.773+0000 7f09d9ffb700 1 -- 172.21.15.190:0/2294484973 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09dc041000 con 0x7f09e80f4740 2024-01-16T17:13:48.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.773+0000 7f09eccb5700 1 --2- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f09d402ff60 0x7f09d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:48.755 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.774+0000 7f09eccb5700 1 --2- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f09d402ff60 0x7f09d4032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f09e40096a0 tx=0x7f09e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:48.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.778+0000 7f09d9ffb700 1 -- 172.21.15.190:0/2294484973 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09dc01d7f0 con 0x7f09e80f4740 2024-01-16T17:13:48.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.985+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09e8108100 con 0x7f09e80f4740 2024-01-16T17:13:48.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.986+0000 7f09d9ffb700 1 -- 172.21.15.190:0/2294484973 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f09dc010230 con 0x7f09e80f4740 2024-01-16T17:13:48.966 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:48.966 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:48.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f09d402ff60 msgr2=0x7f09d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:48.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 --2- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f09d402ff60 0x7f09d4032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f09e40096a0 tx=0x7f09e4006b40).stop 2024-01-16T17:13:48.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 msgr2=0x7f09e81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:48.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 --2- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 0x7f09e81066d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f09dc013510 tx=0x7f09dc00b040).stop 2024-01-16T17:13:48.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 shutdown_connections 2024-01-16T17:13:48.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 --2- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f09d402ff60 0x7f09d4032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:48.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 --2- 172.21.15.190:0/2294484973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e80f4740 0x7f09e81066d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:48.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 >> 172.21.15.190:0/2294484973 conn(0x7f09e80f0260 msgr2=0x7f09e80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:48.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 shutdown_connections 2024-01-16T17:13:48.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:48.988+0000 7f09ef71a700 1 -- 172.21.15.190:0/2294484973 wait complete. 2024-01-16T17:13:48.969 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:49.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:47.810207+0000 mgr.smithi154.sllivz (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:49.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:48.987295+0000 mon.smithi154 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.190:0/2294484973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:50.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:50.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:51.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.469+0000 7f1263fff700 1 -- 172.21.15.190:0/3040000936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 msgr2=0x7f12640f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:51.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.469+0000 7f1263fff700 1 --2- 172.21.15.190:0/3040000936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 0x7f12640f8fc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1254009fb0 tx=0x7f1254009b30).stop 2024-01-16T17:13:51.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.469+0000 7f1263fff700 1 -- 172.21.15.190:0/3040000936 shutdown_connections 2024-01-16T17:13:51.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.469+0000 7f1263fff700 1 --2- 172.21.15.190:0/3040000936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 0x7f12640f8fc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:51.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.469+0000 7f1263fff700 1 -- 172.21.15.190:0/3040000936 >> 172.21.15.190:0/3040000936 conn(0x7f12640f09a0 msgr2=0x7f12640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:51.452 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.469+0000 7f1263fff700 1 -- 172.21.15.190:0/3040000936 shutdown_connections 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.469+0000 7f1263fff700 1 -- 172.21.15.190:0/3040000936 wait complete. 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.470+0000 7f1263fff700 1 Processor -- start 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.470+0000 7f1263fff700 1 -- start start 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.470+0000 7f1263fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 0x7f1264106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.470+0000 7f1263fff700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1264106a10 con 0x7f1264060140 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.470+0000 7f1262ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 0x7f1264106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.470+0000 7f1262ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 0x7f1264106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:41820/0 (socket says 172.21.15.190:41820) 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.470+0000 7f1262ffd700 1 -- 172.21.15.190:0/302775330 learned_addr learned my addr 172.21.15.190:0/302775330 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.471+0000 7f1262ffd700 1 -- 172.21.15.190:0/302775330 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1254005040 con 0x7f1264060140 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.471+0000 7f1262ffd700 1 --2- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 0x7f1264106520 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1254013510 tx=0x7f125400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.471+0000 7f124b7fe700 1 -- 172.21.15.190:0/302775330 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1254009320 con 0x7f1264060140 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.471+0000 7f124b7fe700 1 -- 172.21.15.190:0/302775330 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1254013f00 con 0x7f1264060140 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.471+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1264106c10 con 0x7f1264060140 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.471+0000 7f124b7fe700 1 -- 172.21.15.190:0/302775330 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1254014d90 con 0x7f1264060140 2024-01-16T17:13:51.453 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.471+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12641070a0 con 0x7f1264060140 2024-01-16T17:13:51.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.472+0000 7f124b7fe700 1 -- 172.21.15.190:0/302775330 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12540095f0 con 0x7f1264060140 2024-01-16T17:13:51.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.472+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1264045ad0 con 0x7f1264060140 2024-01-16T17:13:51.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.472+0000 7f124b7fe700 1 --2- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f124c02ff60 0x7f124c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:51.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.472+0000 7f124b7fe700 1 -- 172.21.15.190:0/302775330 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1254041000 con 0x7f1264060140 2024-01-16T17:13:51.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.472+0000 7f12627fc700 1 --2- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f124c02ff60 0x7f124c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:51.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.473+0000 7f12627fc700 1 --2- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f124c02ff60 0x7f124c032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f12580096a0 tx=0x7f1258006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:51.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.477+0000 7f124b7fe700 1 -- 172.21.15.190:0/302775330 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1254014420 con 0x7f1264060140 2024-01-16T17:13:51.665 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.684+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1264107f50 con 0x7f1264060140 2024-01-16T17:13:51.665 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.684+0000 7f124b7fe700 1 -- 172.21.15.190:0/302775330 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f125401d7f0 con 0x7f1264060140 2024-01-16T17:13:51.665 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:51.665 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f124c02ff60 msgr2=0x7f124c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 --2- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f124c02ff60 0x7f124c032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f12580096a0 tx=0x7f1258006b40).stop 2024-01-16T17:13:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 msgr2=0x7f1264106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 --2- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 0x7f1264106520 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1254013510 tx=0x7f125400b040).stop 2024-01-16T17:13:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 shutdown_connections 2024-01-16T17:13:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 --2- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f124c02ff60 0x7f124c032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 --2- 172.21.15.190:0/302775330 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1264060140 0x7f1264106520 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:51.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 >> 172.21.15.190:0/302775330 conn(0x7f12640f09a0 msgr2=0x7f12640f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:51.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 shutdown_connections 2024-01-16T17:13:51.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:51.686+0000 7f1263fff700 1 -- 172.21.15.190:0/302775330 wait complete. 2024-01-16T17:13:51.668 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:51.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:49.810492+0000 mgr.smithi154.sllivz (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:52.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:52 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:51.686359+0000 mon.smithi154 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.190:0/302775330' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:53.053 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:53.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:53.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:51.810863+0000 mgr.smithi154.sllivz (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:54.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.218+0000 7f60b2d6e700 1 -- 172.21.15.190:0/1969167769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 msgr2=0x7f60ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:54.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.218+0000 7f60b2d6e700 1 --2- 172.21.15.190:0/1969167769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 0x7f60ac0f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f60a0009fb0 tx=0x7f60a0009b30).stop 2024-01-16T17:13:54.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.218+0000 7f60b2d6e700 1 -- 172.21.15.190:0/1969167769 shutdown_connections 2024-01-16T17:13:54.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.218+0000 7f60b2d6e700 1 --2- 172.21.15.190:0/1969167769 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 0x7f60ac0f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:54.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.218+0000 7f60b2d6e700 1 -- 172.21.15.190:0/1969167769 >> 172.21.15.190:0/1969167769 conn(0x7f60ac0f0b70 msgr2=0x7f60ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:54.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.218+0000 7f60b2d6e700 1 -- 172.21.15.190:0/1969167769 shutdown_connections 2024-01-16T17:13:54.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.218+0000 7f60b2d6e700 1 -- 172.21.15.190:0/1969167769 wait complete. 2024-01-16T17:13:54.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.219+0000 7f60b2d6e700 1 Processor -- start 2024-01-16T17:13:54.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.219+0000 7f60b2d6e700 1 -- start start 2024-01-16T17:13:54.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.219+0000 7f60b2d6e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 0x7f60ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:54.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.219+0000 7f60b2d6e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac106c20 con 0x7f60ac0f5410 2024-01-16T17:13:54.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60b0b0a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 0x7f60ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:54.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60b0b0a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 0x7f60ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:40986/0 (socket says 172.21.15.190:40986) 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60b0b0a700 1 -- 172.21.15.190:0/2069191183 learned_addr learned my addr 172.21.15.190:0/2069191183 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60b0b0a700 1 -- 172.21.15.190:0/2069191183 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a0005040 con 0x7f60ac0f5410 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60b0b0a700 1 --2- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 0x7f60ac106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f60a0013510 tx=0x7f60a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60a97fa700 1 -- 172.21.15.190:0/2069191183 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60a0009320 con 0x7f60ac0f5410 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60a97fa700 1 -- 172.21.15.190:0/2069191183 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60a0013f00 con 0x7f60ac0f5410 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ac106e20 con 0x7f60ac0f5410 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60a97fa700 1 -- 172.21.15.190:0/2069191183 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60a0014d90 con 0x7f60ac0f5410 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.220+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ac1072b0 con 0x7f60ac0f5410 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.221+0000 7f60a97fa700 1 -- 172.21.15.190:0/2069191183 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60a0014420 con 0x7f60ac0f5410 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.221+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ac045ad0 con 0x7f60ac0f5410 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.221+0000 7f60a97fa700 1 --2- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f609c02ffb0 0x7f609c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:54.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.221+0000 7f60a97fa700 1 -- 172.21.15.190:0/2069191183 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60a00410c0 con 0x7f60ac0f5410 2024-01-16T17:13:54.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.222+0000 7f60abfff700 1 --2- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f609c02ffb0 0x7f609c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:54.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.222+0000 7f60abfff700 1 --2- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f609c02ffb0 0x7f609c032470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f60980096a0 tx=0x7f6098006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:54.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.226+0000 7f60a97fa700 1 -- 172.21.15.190:0/2069191183 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60a001dd50 con 0x7f60ac0f5410 2024-01-16T17:13:54.413 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.431+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60ac0fae60 con 0x7f60ac0f5410 2024-01-16T17:13:54.413 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.432+0000 7f60a97fa700 1 -- 172.21.15.190:0/2069191183 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f60a001db70 con 0x7f60ac0f5410 2024-01-16T17:13:54.413 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:54.414 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:54.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.433+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f609c02ffb0 msgr2=0x7f609c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:54.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.433+0000 7f60b2d6e700 1 --2- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f609c02ffb0 0x7f609c032470 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f60980096a0 tx=0x7f6098006b40).stop 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.433+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 msgr2=0x7f60ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.433+0000 7f60b2d6e700 1 --2- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 0x7f60ac106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f60a0013510 tx=0x7f60a000b040).stop 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.433+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 shutdown_connections 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.433+0000 7f60b2d6e700 1 --2- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f609c02ffb0 0x7f609c032470 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.433+0000 7f60b2d6e700 1 --2- 172.21.15.190:0/2069191183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60ac0f5410 0x7f60ac106730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.433+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 >> 172.21.15.190:0/2069191183 conn(0x7f60ac0f0b70 msgr2=0x7f60ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.434+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 shutdown_connections 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:54.434+0000 7f60b2d6e700 1 -- 172.21.15.190:0/2069191183 wait complete. 2024-01-16T17:13:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:54.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:54 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:54.433963+0000 mon.smithi154 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.190:0/2069191183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:55.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:55.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:55.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:53.811204+0000 mgr.smithi154.sllivz (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:56.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.959+0000 7f5d4e432700 1 -- 172.21.15.190:0/917737229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 msgr2=0x7f5d480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:56.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.959+0000 7f5d4e432700 1 --2- 172.21.15.190:0/917737229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 0x7f5d480f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f5d38009fb0 tx=0x7f5d38009b30).stop 2024-01-16T17:13:56.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.959+0000 7f5d4e432700 1 -- 172.21.15.190:0/917737229 shutdown_connections 2024-01-16T17:13:56.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.959+0000 7f5d4e432700 1 --2- 172.21.15.190:0/917737229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 0x7f5d480f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:56.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.959+0000 7f5d4e432700 1 -- 172.21.15.190:0/917737229 >> 172.21.15.190:0/917737229 conn(0x7f5d480f1c70 msgr2=0x7f5d480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:56.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.959+0000 7f5d4e432700 1 -- 172.21.15.190:0/917737229 shutdown_connections 2024-01-16T17:13:56.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.959+0000 7f5d4e432700 1 -- 172.21.15.190:0/917737229 wait complete. 2024-01-16T17:13:56.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.960+0000 7f5d4e432700 1 Processor -- start 2024-01-16T17:13:56.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.960+0000 7f5d4e432700 1 -- start start 2024-01-16T17:13:56.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.960+0000 7f5d4e432700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 0x7f5d48106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:56.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.960+0000 7f5d4e432700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d48106c20 con 0x7f5d480f6510 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d47fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 0x7f5d48106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d47fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 0x7f5d48106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:40998/0 (socket says 172.21.15.190:40998) 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d47fff700 1 -- 172.21.15.190:0/1784334253 learned_addr learned my addr 172.21.15.190:0/1784334253 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d47fff700 1 -- 172.21.15.190:0/1784334253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d38005040 con 0x7f5d480f6510 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d47fff700 1 --2- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 0x7f5d48106730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f5d38013510 tx=0x7f5d3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d44ff9700 1 -- 172.21.15.190:0/1784334253 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d38009320 con 0x7f5d480f6510 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d48106e20 con 0x7f5d480f6510 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d44ff9700 1 -- 172.21.15.190:0/1784334253 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d38013f00 con 0x7f5d480f6510 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d44ff9700 1 -- 172.21.15.190:0/1784334253 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d38014d90 con 0x7f5d480f6510 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.961+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d481072b0 con 0x7f5d480f6510 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.962+0000 7f5d44ff9700 1 -- 172.21.15.190:0/1784334253 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d380263a0 con 0x7f5d480f6510 2024-01-16T17:13:56.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.962+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d48045ad0 con 0x7f5d480f6510 2024-01-16T17:13:56.945 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.962+0000 7f5d44ff9700 1 --2- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3002ffb0 0x7f5d30032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:56.945 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.962+0000 7f5d44ff9700 1 -- 172.21.15.190:0/1784334253 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d38041050 con 0x7f5d480f6510 2024-01-16T17:13:56.945 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.963+0000 7f5d477fe700 1 --2- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3002ffb0 0x7f5d30032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:56.945 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.963+0000 7f5d477fe700 1 --2- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3002ffb0 0x7f5d30032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f5d3c0096a0 tx=0x7f5d3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:56.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:56.967+0000 7f5d44ff9700 1 -- 172.21.15.190:0/1784334253 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d380126f0 con 0x7f5d480f6510 2024-01-16T17:13:56.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:55.927546+0000 mon.smithi154 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:13:56.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:55.929259+0000 mon.smithi154 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:13:57.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.177+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d480fbf60 con 0x7f5d480f6510 2024-01-16T17:13:57.158 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.177+0000 7f5d44ff9700 1 -- 172.21.15.190:0/1784334253 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d38012330 con 0x7f5d480f6510 2024-01-16T17:13:57.158 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:57.158 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:57.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3002ffb0 msgr2=0x7f5d30032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:57.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 --2- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3002ffb0 0x7f5d30032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f5d3c0096a0 tx=0x7f5d3c006b40).stop 2024-01-16T17:13:57.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 msgr2=0x7f5d48106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:57.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 --2- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 0x7f5d48106730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f5d38013510 tx=0x7f5d3800b040).stop 2024-01-16T17:13:57.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 shutdown_connections 2024-01-16T17:13:57.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 --2- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5d3002ffb0 0x7f5d30032470 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:57.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 --2- 172.21.15.190:0/1784334253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d480f6510 0x7f5d48106730 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:57.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 >> 172.21.15.190:0/1784334253 conn(0x7f5d480f1c70 msgr2=0x7f5d480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:57.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 shutdown_connections 2024-01-16T17:13:57.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:57.179+0000 7f5d4e432700 1 -- 172.21.15.190:0/1784334253 wait complete. 2024-01-16T17:13:57.161 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:57.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:55.811540+0000 mgr.smithi154.sllivz (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:13:57.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:57.179453+0000 mon.smithi154 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.190:0/1784334253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:13:58.540 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:13:58.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:13:59.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.739+0000 7f88d35d9700 1 -- 172.21.15.190:0/3468541959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 msgr2=0x7f88cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:59.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.739+0000 7f88d35d9700 1 --2- 172.21.15.190:0/3468541959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 0x7f88cc0f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f88c0009fb0 tx=0x7f88c0009b30).stop 2024-01-16T17:13:59.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.739+0000 7f88d35d9700 1 -- 172.21.15.190:0/3468541959 shutdown_connections 2024-01-16T17:13:59.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.739+0000 7f88d35d9700 1 --2- 172.21.15.190:0/3468541959 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 0x7f88cc0f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:59.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.739+0000 7f88d35d9700 1 -- 172.21.15.190:0/3468541959 >> 172.21.15.190:0/3468541959 conn(0x7f88cc0f0b70 msgr2=0x7f88cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:59.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d35d9700 1 -- 172.21.15.190:0/3468541959 shutdown_connections 2024-01-16T17:13:59.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d35d9700 1 -- 172.21.15.190:0/3468541959 wait complete. 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d35d9700 1 Processor -- start 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d35d9700 1 -- start start 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d35d9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 0x7f88cc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d35d9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc0fb820 con 0x7f88cc0f5410 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d1375700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 0x7f88cc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d1375700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 0x7f88cc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:41020/0 (socket says 172.21.15.190:41020) 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.740+0000 7f88d1375700 1 -- 172.21.15.190:0/403618425 learned_addr learned my addr 172.21.15.190:0/403618425 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.741+0000 7f88d1375700 1 -- 172.21.15.190:0/403618425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c0005040 con 0x7f88cc0f5410 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.741+0000 7f88d1375700 1 --2- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 0x7f88cc0fd3a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f88c0013510 tx=0x7f88c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.741+0000 7f88bdffb700 1 -- 172.21.15.190:0/403618425 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88c0009320 con 0x7f88cc0f5410 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.741+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88cc0fb9c0 con 0x7f88cc0f5410 2024-01-16T17:13:59.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.741+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88cc0fbe50 con 0x7f88cc0f5410 2024-01-16T17:13:59.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.741+0000 7f88bdffb700 1 -- 172.21.15.190:0/403618425 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88c0013f00 con 0x7f88cc0f5410 2024-01-16T17:13:59.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.741+0000 7f88bdffb700 1 -- 172.21.15.190:0/403618425 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88c0014d60 con 0x7f88cc0f5410 2024-01-16T17:13:59.725 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.742+0000 7f88bdffb700 1 -- 172.21.15.190:0/403618425 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88c00263a0 con 0x7f88cc0f5410 2024-01-16T17:13:59.725 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.742+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88cc059100 con 0x7f88cc0f5410 2024-01-16T17:13:59.725 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.742+0000 7f88bdffb700 1 --2- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f88b802ff60 0x7f88b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:13:59.725 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.742+0000 7f88bdffb700 1 -- 172.21.15.190:0/403618425 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88c00413b0 con 0x7f88cc0f5410 2024-01-16T17:13:59.725 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.742+0000 7f88d0b74700 1 --2- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f88b802ff60 0x7f88b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:13:59.725 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.743+0000 7f88d0b74700 1 --2- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f88b802ff60 0x7f88b8032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f88c80096a0 tx=0x7f88c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:13:59.728 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.748+0000 7f88bdffb700 1 -- 172.21.15.190:0/403618425 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88c0012ae0 con 0x7f88cc0f5410 2024-01-16T17:13:59.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.952+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88cc0fae60 con 0x7f88cc0f5410 2024-01-16T17:13:59.934 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.954+0000 7f88bdffb700 1 -- 172.21.15.190:0/403618425 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f88c0012ae0 con 0x7f88cc0f5410 2024-01-16T17:13:59.934 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:13:59.934 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.954+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f88b802ff60 msgr2=0x7f88b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.954+0000 7f88d35d9700 1 --2- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f88b802ff60 0x7f88b8032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f88c80096a0 tx=0x7f88c8006b40).stop 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.954+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 msgr2=0x7f88cc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.954+0000 7f88d35d9700 1 --2- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 0x7f88cc0fd3a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f88c0013510 tx=0x7f88c000b040).stop 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.955+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 shutdown_connections 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.955+0000 7f88d35d9700 1 --2- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f88b802ff60 0x7f88b8032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.955+0000 7f88d35d9700 1 --2- 172.21.15.190:0/403618425 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88cc0f5410 0x7f88cc0fd3a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.955+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 >> 172.21.15.190:0/403618425 conn(0x7f88cc0f0b70 msgr2=0x7f88cc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.955+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 shutdown_connections 2024-01-16T17:13:59.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:13:59.955+0000 7f88d35d9700 1 -- 172.21.15.190:0/403618425 wait complete. 2024-01-16T17:13:59.937 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:13:59.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:13:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:57.811847+0000 mgr.smithi154.sllivz (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:00.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:00 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:13:59.954620+0000 mon.smithi154 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.190:0/403618425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:01.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:01.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:01.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:13:59.812215+0000 mgr.smithi154.sllivz (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:02.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.397+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2491785331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 msgr2=0x7ffbb00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:02.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.397+0000 7ffbb6d56700 1 --2- 172.21.15.190:0/2491785331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 0x7ffbb00f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7ffba4009fb0 tx=0x7ffba4009b30).stop 2024-01-16T17:14:02.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.397+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2491785331 shutdown_connections 2024-01-16T17:14:02.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.397+0000 7ffbb6d56700 1 --2- 172.21.15.190:0/2491785331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 0x7ffbb00f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:02.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.397+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2491785331 >> 172.21.15.190:0/2491785331 conn(0x7ffbb00f1c70 msgr2=0x7ffbb00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:02.378 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.397+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2491785331 shutdown_connections 2024-01-16T17:14:02.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.397+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2491785331 wait complete. 2024-01-16T17:14:02.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.398+0000 7ffbb6d56700 1 Processor -- start 2024-01-16T17:14:02.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.398+0000 7ffbb6d56700 1 -- start start 2024-01-16T17:14:02.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.398+0000 7ffbb6d56700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 0x7ffbb006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:02.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.398+0000 7ffbb6d56700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0069e20 con 0x7ffbb00f6510 2024-01-16T17:14:02.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.398+0000 7ffbb4af2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 0x7ffbb006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:02.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.398+0000 7ffbb4af2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 0x7ffbb006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50342/0 (socket says 172.21.15.190:50342) 2024-01-16T17:14:02.379 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.398+0000 7ffbb4af2700 1 -- 172.21.15.190:0/2920367038 learned_addr learned my addr 172.21.15.190:0/2920367038 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:02.380 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.399+0000 7ffbb4af2700 1 -- 172.21.15.190:0/2920367038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba4005040 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.399+0000 7ffbb4af2700 1 --2- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 0x7ffbb006b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ffba4013510 tx=0x7ffba400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.399+0000 7ffbad7fa700 1 -- 172.21.15.190:0/2920367038 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffba4009320 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.399+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb0069fc0 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.399+0000 7ffbad7fa700 1 -- 172.21.15.190:0/2920367038 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba4013f00 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.399+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb006a450 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.399+0000 7ffbad7fa700 1 -- 172.21.15.190:0/2920367038 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffba4014d60 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.400+0000 7ffbad7fa700 1 -- 172.21.15.190:0/2920367038 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffba40263a0 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.400+0000 7ffbad7fa700 1 --2- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffba002ff60 0x7ffba0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.400+0000 7ffbad7fa700 1 -- 172.21.15.190:0/2920367038 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffba40095f0 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.400+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb0059100 con 0x7ffbb00f6510 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.400+0000 7ffbaffff700 1 --2- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffba002ff60 0x7ffba0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:02.381 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.400+0000 7ffbaffff700 1 --2- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffba002ff60 0x7ffba0032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ffb9c0096a0 tx=0x7ffb9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:02.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.405+0000 7ffbad7fa700 1 -- 172.21.15.190:0/2920367038 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffba401d380 con 0x7ffbb00f6510 2024-01-16T17:14:02.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.613+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbb0045ad0 con 0x7ffbb00f6510 2024-01-16T17:14:02.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.614+0000 7ffbad7fa700 1 -- 172.21.15.190:0/2920367038 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffba402b020 con 0x7ffbb00f6510 2024-01-16T17:14:02.594 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:02.594 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:02.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.615+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffba002ff60 msgr2=0x7ffba0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:02.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.615+0000 7ffbb6d56700 1 --2- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffba002ff60 0x7ffba0032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ffb9c0096a0 tx=0x7ffb9c006b40).stop 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.615+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 msgr2=0x7ffbb006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.615+0000 7ffbb6d56700 1 --2- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 0x7ffbb006b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ffba4013510 tx=0x7ffba400b040).stop 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.615+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 shutdown_connections 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.615+0000 7ffbb6d56700 1 --2- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffba002ff60 0x7ffba0032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.615+0000 7ffbb6d56700 1 --2- 172.21.15.190:0/2920367038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbb00f6510 0x7ffbb006b9a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.615+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 >> 172.21.15.190:0/2920367038 conn(0x7ffbb00f1c70 msgr2=0x7ffbb00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.616+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 shutdown_connections 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:02.616+0000 7ffbb6d56700 1 -- 172.21.15.190:0/2920367038 wait complete. 2024-01-16T17:14:02.596 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:03.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:01.812570+0000 mgr.smithi154.sllivz (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:03.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:02.614850+0000 mon.smithi154 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.190:0/2920367038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:03.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:03.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.106+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/2035939965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 msgr2=0x7f0f980f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.106+0000 7f0f9e3a7700 1 --2- 172.21.15.190:0/2035939965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 0x7f0f980f9150 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f0f88009fb0 tx=0x7f0f88009b30).stop 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.106+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/2035939965 shutdown_connections 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.106+0000 7f0f9e3a7700 1 --2- 172.21.15.190:0/2035939965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 0x7f0f980f9150 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.106+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/2035939965 >> 172.21.15.190:0/2035939965 conn(0x7f0f980f0b70 msgr2=0x7f0f980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.106+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/2035939965 shutdown_connections 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.106+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/2035939965 wait complete. 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.107+0000 7f0f9e3a7700 1 Processor -- start 2024-01-16T17:14:05.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.107+0000 7f0f9e3a7700 1 -- start start 2024-01-16T17:14:05.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.107+0000 7f0f9e3a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 0x7f0f981066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:05.088 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.107+0000 7f0f9e3a7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98106b90 con 0x7f0f98060220 2024-01-16T17:14:05.089 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 0x7f0f981066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:05.089 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f97fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 0x7f0f981066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50358/0 (socket says 172.21.15.190:50358) 2024-01-16T17:14:05.089 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f97fff700 1 -- 172.21.15.190:0/778261216 learned_addr learned my addr 172.21.15.190:0/778261216 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f97fff700 1 -- 172.21.15.190:0/778261216 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f88005040 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f97fff700 1 --2- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 0x7f0f981066a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f0f88013510 tx=0x7f0f8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f94ff9700 1 -- 172.21.15.190:0/778261216 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f88009320 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f94ff9700 1 -- 172.21.15.190:0/778261216 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f88013f00 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f98106d90 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f98107220 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.108+0000 7f0f94ff9700 1 -- 172.21.15.190:0/778261216 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f88014dc0 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.109+0000 7f0f94ff9700 1 -- 172.21.15.190:0/778261216 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f8801d3d0 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.109+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f98045ad0 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.109+0000 7f0f94ff9700 1 --2- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f8002ff60 0x7f0f80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.109+0000 7f0f94ff9700 1 -- 172.21.15.190:0/778261216 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f88041410 con 0x7f0f98060220 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.109+0000 7f0f977fe700 1 --2- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f8002ff60 0x7f0f80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:05.090 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.110+0000 7f0f977fe700 1 --2- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f8002ff60 0x7f0f80032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0f8c0096a0 tx=0x7f0f8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:05.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.114+0000 7f0f94ff9700 1 -- 172.21.15.190:0/778261216 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f88012510 con 0x7f0f98060220 2024-01-16T17:14:05.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.318+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f981080d0 con 0x7f0f98060220 2024-01-16T17:14:05.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.319+0000 7f0f94ff9700 1 -- 172.21.15.190:0/778261216 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f88012330 con 0x7f0f98060220 2024-01-16T17:14:05.299 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:05.299 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:05.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f8002ff60 msgr2=0x7f0f80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 --2- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f8002ff60 0x7f0f80032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f0f8c0096a0 tx=0x7f0f8c006b40).stop 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 msgr2=0x7f0f981066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 --2- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 0x7f0f981066a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f0f88013510 tx=0x7f0f8800b040).stop 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 shutdown_connections 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 --2- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f8002ff60 0x7f0f80032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 --2- 172.21.15.190:0/778261216 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f98060220 0x7f0f981066a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 >> 172.21.15.190:0/778261216 conn(0x7f0f980f0b70 msgr2=0x7f0f980f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 shutdown_connections 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:05.321+0000 7f0f9e3a7700 1 -- 172.21.15.190:0/778261216 wait complete. 2024-01-16T17:14:05.301 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:05.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:03.812844+0000 mgr.smithi154.sllivz (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:05.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:05.319839+0000 mon.smithi154 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.190:0/778261216' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:06.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:06.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:07.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 -- 172.21.15.190:0/2817026224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 msgr2=0x7f19340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 --2- 172.21.15.190:0/2817026224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 0x7f19340f90f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1928009fb0 tx=0x7f1928009b30).stop 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 -- 172.21.15.190:0/2817026224 shutdown_connections 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 --2- 172.21.15.190:0/2817026224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 0x7f19340f90f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 -- 172.21.15.190:0/2817026224 >> 172.21.15.190:0/2817026224 conn(0x7f19340f0b70 msgr2=0x7f19340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 -- 172.21.15.190:0/2817026224 shutdown_connections 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 -- 172.21.15.190:0/2817026224 wait complete. 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 Processor -- start 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.891+0000 7f193a07e700 1 -- start start 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f193a07e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 0x7f19341088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f193a07e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934108dc0 con 0x7f1934060280 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f19337fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 0x7f19341088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f19337fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 0x7f19341088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50384/0 (socket says 172.21.15.190:50384) 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f19337fe700 1 -- 172.21.15.190:0/2852160205 learned_addr learned my addr 172.21.15.190:0/2852160205 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:07.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f19337fe700 1 -- 172.21.15.190:0/2852160205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1928005040 con 0x7f1934060280 2024-01-16T17:14:07.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f19337fe700 1 --2- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 0x7f19341088d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1928013510 tx=0x7f192800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:07.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f191bfff700 1 -- 172.21.15.190:0/2852160205 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1928009320 con 0x7f1934060280 2024-01-16T17:14:07.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1934108fc0 con 0x7f1934060280 2024-01-16T17:14:07.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f191bfff700 1 -- 172.21.15.190:0/2852160205 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1928013f00 con 0x7f1934060280 2024-01-16T17:14:07.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f191bfff700 1 -- 172.21.15.190:0/2852160205 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1928014dc0 con 0x7f1934060280 2024-01-16T17:14:07.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.892+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1934109450 con 0x7f1934060280 2024-01-16T17:14:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.893+0000 7f191bfff700 1 -- 172.21.15.190:0/2852160205 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1928014420 con 0x7f1934060280 2024-01-16T17:14:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.893+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1934045ad0 con 0x7f1934060280 2024-01-16T17:14:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.893+0000 7f191bfff700 1 --2- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f191c02ff60 0x7f191c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.893+0000 7f191bfff700 1 -- 172.21.15.190:0/2852160205 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1928041040 con 0x7f1934060280 2024-01-16T17:14:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.893+0000 7f1932ffd700 1 --2- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f191c02ff60 0x7f191c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:07.875 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.894+0000 7f1932ffd700 1 --2- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f191c02ff60 0x7f191c032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f19240096a0 tx=0x7f1924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:07.877 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:07.898+0000 7f191bfff700 1 -- 172.21.15.190:0/2852160205 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19280095f0 con 0x7f1934060280 2024-01-16T17:14:07.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:05.813163+0000 mgr.smithi154.sllivz (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:08.083 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.103+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f193410a2a0 con 0x7f1934060280 2024-01-16T17:14:08.083 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.104+0000 7f191bfff700 1 -- 172.21.15.190:0/2852160205 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f192801cb10 con 0x7f1934060280 2024-01-16T17:14:08.084 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:08.084 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f191c02ff60 msgr2=0x7f191c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 --2- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f191c02ff60 0x7f191c032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f19240096a0 tx=0x7f1924006b40).stop 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 msgr2=0x7f19341088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 --2- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 0x7f19341088d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1928013510 tx=0x7f192800b040).stop 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 shutdown_connections 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 --2- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f191c02ff60 0x7f191c032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 --2- 172.21.15.190:0/2852160205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1934060280 0x7f19341088d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 >> 172.21.15.190:0/2852160205 conn(0x7f19340f0b70 msgr2=0x7f19340f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 shutdown_connections 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:08.105+0000 7f193a07e700 1 -- 172.21.15.190:0/2852160205 wait complete. 2024-01-16T17:14:08.086 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:08.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:08.104820+0000 mon.smithi154 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.190:0/2852160205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:09.453 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:09.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:09.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:07.813471+0000 mgr.smithi154.sllivz (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:10.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.608+0000 7f95d5b51700 1 -- 172.21.15.190:0/4064487141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 msgr2=0x7f95d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:10.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.608+0000 7f95d5b51700 1 --2- 172.21.15.190:0/4064487141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 0x7f95d00f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f95c0009fb0 tx=0x7f95c0009b30).stop 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.608+0000 7f95d5b51700 1 -- 172.21.15.190:0/4064487141 shutdown_connections 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.608+0000 7f95d5b51700 1 --2- 172.21.15.190:0/4064487141 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 0x7f95d00f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.608+0000 7f95d5b51700 1 -- 172.21.15.190:0/4064487141 >> 172.21.15.190:0/4064487141 conn(0x7f95d00f0b70 msgr2=0x7f95d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.608+0000 7f95d5b51700 1 -- 172.21.15.190:0/4064487141 shutdown_connections 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.608+0000 7f95d5b51700 1 -- 172.21.15.190:0/4064487141 wait complete. 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.609+0000 7f95d5b51700 1 Processor -- start 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.609+0000 7f95d5b51700 1 -- start start 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.610+0000 7f95d5b51700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 0x7f95d0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:10.590 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.610+0000 7f95d5b51700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d0105b20 con 0x7f95d00f5410 2024-01-16T17:14:10.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.610+0000 7f95cf7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 0x7f95d0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.610+0000 7f95cf7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 0x7f95d0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50396/0 (socket says 172.21.15.190:50396) 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.610+0000 7f95cf7fe700 1 -- 172.21.15.190:0/2608439010 learned_addr learned my addr 172.21.15.190:0/2608439010 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.610+0000 7f95cf7fe700 1 -- 172.21.15.190:0/2608439010 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c0005040 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.610+0000 7f95cf7fe700 1 --2- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 0x7f95d0105630 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f95c0013510 tx=0x7f95c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.611+0000 7f95b7fff700 1 -- 172.21.15.190:0/2608439010 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95c0009320 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.611+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d0105d20 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.611+0000 7f95b7fff700 1 -- 172.21.15.190:0/2608439010 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95c0013f00 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.611+0000 7f95b7fff700 1 -- 172.21.15.190:0/2608439010 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95c0014d90 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.611+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d01061b0 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.612+0000 7f95b7fff700 1 -- 172.21.15.190:0/2608439010 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95c00095f0 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.612+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d0045ad0 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.612+0000 7f95b7fff700 1 --2- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95b802ff60 0x7f95b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.612+0000 7f95b7fff700 1 -- 172.21.15.190:0/2608439010 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95c0040fe0 con 0x7f95d00f5410 2024-01-16T17:14:10.592 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.612+0000 7f95ceffd700 1 --2- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95b802ff60 0x7f95b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:10.593 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.612+0000 7f95ceffd700 1 --2- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95b802ff60 0x7f95b8032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f95c40096a0 tx=0x7f95c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:10.596 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.617+0000 7f95b7fff700 1 -- 172.21.15.190:0/2608439010 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95c001d7f0 con 0x7f95d00f5410 2024-01-16T17:14:10.802 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.822+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95d00fae60 con 0x7f95d00f5410 2024-01-16T17:14:10.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.823+0000 7f95b7fff700 1 -- 172.21.15.190:0/2608439010 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f95c0010230 con 0x7f95d00f5410 2024-01-16T17:14:10.803 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:10.803 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95b802ff60 msgr2=0x7f95b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 --2- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95b802ff60 0x7f95b8032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f95c40096a0 tx=0x7f95c4006b40).stop 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 msgr2=0x7f95d0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 --2- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 0x7f95d0105630 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f95c0013510 tx=0x7f95c000b040).stop 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 shutdown_connections 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 --2- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f95b802ff60 0x7f95b8032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 --2- 172.21.15.190:0/2608439010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f95d00f5410 0x7f95d0105630 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 >> 172.21.15.190:0/2608439010 conn(0x7f95d00f0b70 msgr2=0x7f95d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 shutdown_connections 2024-01-16T17:14:10.805 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:10.825+0000 7f95d5b51700 1 -- 172.21.15.190:0/2608439010 wait complete. 2024-01-16T17:14:10.806 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:11.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:09.813816+0000 mgr.smithi154.sllivz (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:11.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:10.823932+0000 mon.smithi154 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.190:0/2608439010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:12.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:12.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.262+0000 7f2a4b309700 1 -- 172.21.15.190:0/1204634069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 msgr2=0x7f2a440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.262+0000 7f2a4b309700 1 --2- 172.21.15.190:0/1204634069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 0x7f2a440f6930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f2a34009fb0 tx=0x7f2a34009b30).stop 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.262+0000 7f2a4b309700 1 -- 172.21.15.190:0/1204634069 shutdown_connections 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.262+0000 7f2a4b309700 1 --2- 172.21.15.190:0/1204634069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 0x7f2a440f6930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.262+0000 7f2a4b309700 1 -- 172.21.15.190:0/1204634069 >> 172.21.15.190:0/1204634069 conn(0x7f2a440f1c70 msgr2=0x7f2a440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.262+0000 7f2a4b309700 1 -- 172.21.15.190:0/1204634069 shutdown_connections 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.262+0000 7f2a4b309700 1 -- 172.21.15.190:0/1204634069 wait complete. 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.263+0000 7f2a4b309700 1 Processor -- start 2024-01-16T17:14:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.263+0000 7f2a4b309700 1 -- start start 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.263+0000 7f2a4b309700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 0x7f2a44106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.263+0000 7f2a4b309700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a44106c20 con 0x7f2a440f6510 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.263+0000 7f2a490a5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 0x7f2a44106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.263+0000 7f2a490a5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 0x7f2a44106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46896/0 (socket says 172.21.15.190:46896) 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.263+0000 7f2a490a5700 1 -- 172.21.15.190:0/2695917433 learned_addr learned my addr 172.21.15.190:0/2695917433 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.264+0000 7f2a490a5700 1 -- 172.21.15.190:0/2695917433 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34005040 con 0x7f2a440f6510 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.264+0000 7f2a490a5700 1 --2- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 0x7f2a44106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f2a34013510 tx=0x7f2a3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.264+0000 7f2a39ffb700 1 -- 172.21.15.190:0/2695917433 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a34009320 con 0x7f2a440f6510 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.264+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a44106e20 con 0x7f2a440f6510 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.264+0000 7f2a39ffb700 1 -- 172.21.15.190:0/2695917433 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a34013f00 con 0x7f2a440f6510 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.264+0000 7f2a39ffb700 1 -- 172.21.15.190:0/2695917433 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a34014d90 con 0x7f2a440f6510 2024-01-16T17:14:13.245 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.264+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a441072b0 con 0x7f2a440f6510 2024-01-16T17:14:13.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.265+0000 7f2a39ffb700 1 -- 172.21.15.190:0/2695917433 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a340263a0 con 0x7f2a440f6510 2024-01-16T17:14:13.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.265+0000 7f2a39ffb700 1 --2- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a3002ff60 0x7f2a30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:13.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.265+0000 7f2a39ffb700 1 -- 172.21.15.190:0/2695917433 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a34040ff0 con 0x7f2a440f6510 2024-01-16T17:14:13.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.265+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a44045ad0 con 0x7f2a440f6510 2024-01-16T17:14:13.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.265+0000 7f2a488a4700 1 --2- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a3002ff60 0x7f2a30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:13.246 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.266+0000 7f2a488a4700 1 --2- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a3002ff60 0x7f2a30032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2a400096a0 tx=0x7f2a40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:13.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.270+0000 7f2a39ffb700 1 -- 172.21.15.190:0/2695917433 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a34012920 con 0x7f2a440f6510 2024-01-16T17:14:13.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.475+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a44107ff0 con 0x7f2a440f6510 2024-01-16T17:14:13.455 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.476+0000 7f2a39ffb700 1 -- 172.21.15.190:0/2695917433 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2a3402b020 con 0x7f2a440f6510 2024-01-16T17:14:13.456 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:13.456 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a3002ff60 msgr2=0x7f2a30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 --2- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a3002ff60 0x7f2a30032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f2a400096a0 tx=0x7f2a40006b40).stop 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 msgr2=0x7f2a44106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 --2- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 0x7f2a44106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f2a34013510 tx=0x7f2a3400b040).stop 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 shutdown_connections 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 --2- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a3002ff60 0x7f2a30032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 --2- 172.21.15.190:0/2695917433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a440f6510 0x7f2a44106730 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 >> 172.21.15.190:0/2695917433 conn(0x7f2a440f1c70 msgr2=0x7f2a440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 shutdown_connections 2024-01-16T17:14:13.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:13.478+0000 7f2a4b309700 1 -- 172.21.15.190:0/2695917433 wait complete. 2024-01-16T17:14:13.459 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:13.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:11.814127+0000 mgr.smithi154.sllivz (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:13.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:13.476902+0000 mon.smithi154 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.190:0/2695917433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:14.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:14.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:15.935 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.954+0000 7effedbe0700 1 -- 172.21.15.190:0/2721752948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 msgr2=0x7effe80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.954+0000 7effedbe0700 1 --2- 172.21.15.190:0/2721752948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 0x7effe80f6930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7effdc009fb0 tx=0x7effdc009b30).stop 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.954+0000 7effedbe0700 1 -- 172.21.15.190:0/2721752948 shutdown_connections 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.954+0000 7effedbe0700 1 --2- 172.21.15.190:0/2721752948 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 0x7effe80f6930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.954+0000 7effedbe0700 1 -- 172.21.15.190:0/2721752948 >> 172.21.15.190:0/2721752948 conn(0x7effe80f1c70 msgr2=0x7effe80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.954+0000 7effedbe0700 1 -- 172.21.15.190:0/2721752948 shutdown_connections 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.954+0000 7effedbe0700 1 -- 172.21.15.190:0/2721752948 wait complete. 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effedbe0700 1 Processor -- start 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effedbe0700 1 -- start start 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effedbe0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 0x7effe8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effedbe0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe8106c20 con 0x7effe80f6510 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effe77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 0x7effe8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effe77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 0x7effe8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46906/0 (socket says 172.21.15.190:46906) 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effe77fe700 1 -- 172.21.15.190:0/3784402132 learned_addr learned my addr 172.21.15.190:0/3784402132 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:15.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effe77fe700 1 -- 172.21.15.190:0/3784402132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc005040 con 0x7effe80f6510 2024-01-16T17:14:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.955+0000 7effe77fe700 1 --2- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 0x7effe8106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7effdc013510 tx=0x7effdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.956+0000 7effcffff700 1 -- 172.21.15.190:0/3784402132 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effdc009320 con 0x7effe80f6510 2024-01-16T17:14:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.956+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe8106e20 con 0x7effe80f6510 2024-01-16T17:14:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.956+0000 7effcffff700 1 -- 172.21.15.190:0/3784402132 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7effdc013f00 con 0x7effe80f6510 2024-01-16T17:14:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.956+0000 7effcffff700 1 -- 172.21.15.190:0/3784402132 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effdc014d90 con 0x7effe80f6510 2024-01-16T17:14:15.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.956+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe81072b0 con 0x7effe80f6510 2024-01-16T17:14:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.956+0000 7effcffff700 1 -- 172.21.15.190:0/3784402132 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effdc014420 con 0x7effe80f6510 2024-01-16T17:14:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.957+0000 7effcffff700 1 --2- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7effd002ff60 0x7effd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.957+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe8045ad0 con 0x7effe80f6510 2024-01-16T17:14:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.957+0000 7effcffff700 1 -- 172.21.15.190:0/3784402132 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effdc041000 con 0x7effe80f6510 2024-01-16T17:14:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.957+0000 7effe6ffd700 1 --2- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7effd002ff60 0x7effd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.957+0000 7effe6ffd700 1 --2- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7effd002ff60 0x7effd0032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7effd80096a0 tx=0x7effd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:15.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:15.962+0000 7effcffff700 1 -- 172.21.15.190:0/3784402132 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effdc0095f0 con 0x7effe80f6510 2024-01-16T17:14:15.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:13.814445+0000 mgr.smithi154.sllivz (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:16.149 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.169+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effe80fc050 con 0x7effe80f6510 2024-01-16T17:14:16.149 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.170+0000 7effcffff700 1 -- 172.21.15.190:0/3784402132 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7effdc01db70 con 0x7effe80f6510 2024-01-16T17:14:16.150 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:16.150 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:16.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7effd002ff60 msgr2=0x7effd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:16.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 --2- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7effd002ff60 0x7effd0032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7effd80096a0 tx=0x7effd8006b40).stop 2024-01-16T17:14:16.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 msgr2=0x7effe8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:16.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 --2- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 0x7effe8106730 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7effdc013510 tx=0x7effdc00b040).stop 2024-01-16T17:14:16.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 shutdown_connections 2024-01-16T17:14:16.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 --2- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7effd002ff60 0x7effd0032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:16.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 --2- 172.21.15.190:0/3784402132 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7effe80f6510 0x7effe8106730 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:16.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 >> 172.21.15.190:0/3784402132 conn(0x7effe80f1c70 msgr2=0x7effe80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:16.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 shutdown_connections 2024-01-16T17:14:16.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:16.171+0000 7effedbe0700 1 -- 172.21.15.190:0/3784402132 wait complete. 2024-01-16T17:14:16.152 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:16.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:16 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:16.170525+0000 mon.smithi154 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.190:0/3784402132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:17.536 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:17.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:17.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:15.814728+0000 mgr.smithi154.sllivz (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:18.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.713+0000 7faa1bbac700 1 -- 172.21.15.190:0/3609666242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 msgr2=0x7faa140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:18.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.713+0000 7faa1bbac700 1 --2- 172.21.15.190:0/3609666242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 0x7faa140f6930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7faa10009fb0 tx=0x7faa10009b30).stop 2024-01-16T17:14:18.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.713+0000 7faa1bbac700 1 -- 172.21.15.190:0/3609666242 shutdown_connections 2024-01-16T17:14:18.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.713+0000 7faa1bbac700 1 --2- 172.21.15.190:0/3609666242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 0x7faa140f6930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:18.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.713+0000 7faa1bbac700 1 -- 172.21.15.190:0/3609666242 >> 172.21.15.190:0/3609666242 conn(0x7faa140f1c70 msgr2=0x7faa140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:18.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.713+0000 7faa1bbac700 1 -- 172.21.15.190:0/3609666242 shutdown_connections 2024-01-16T17:14:18.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.713+0000 7faa1bbac700 1 -- 172.21.15.190:0/3609666242 wait complete. 2024-01-16T17:14:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.714+0000 7faa1bbac700 1 Processor -- start 2024-01-16T17:14:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.714+0000 7faa1bbac700 1 -- start start 2024-01-16T17:14:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.714+0000 7faa1bbac700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 0x7faa14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.714+0000 7faa1bbac700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa14106c20 con 0x7faa140f6510 2024-01-16T17:14:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.714+0000 7faa19948700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 0x7faa14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.714+0000 7faa19948700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 0x7faa14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46918/0 (socket says 172.21.15.190:46918) 2024-01-16T17:14:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.714+0000 7faa19948700 1 -- 172.21.15.190:0/1005384793 learned_addr learned my addr 172.21.15.190:0/1005384793 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.714+0000 7faa19948700 1 -- 172.21.15.190:0/1005384793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10005040 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.715+0000 7faa19948700 1 --2- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 0x7faa14106730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7faa10013510 tx=0x7faa1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.715+0000 7faa0a7fc700 1 -- 172.21.15.190:0/1005384793 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa10009320 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.715+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa14106e20 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.715+0000 7faa0a7fc700 1 -- 172.21.15.190:0/1005384793 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faa10013f00 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.715+0000 7faa0a7fc700 1 -- 172.21.15.190:0/1005384793 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa10014d90 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.715+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa141072b0 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.716+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa14045ad0 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.716+0000 7faa0a7fc700 1 -- 172.21.15.190:0/1005384793 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faa100095f0 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.716+0000 7faa0a7fc700 1 --2- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faa0002ff60 0x7faa00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.716+0000 7faa0a7fc700 1 -- 172.21.15.190:0/1005384793 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faa10041000 con 0x7faa140f6510 2024-01-16T17:14:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.716+0000 7faa19147700 1 --2- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faa0002ff60 0x7faa00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:18.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.717+0000 7faa19147700 1 --2- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faa0002ff60 0x7faa00032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7faa040096a0 tx=0x7faa04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:18.700 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.720+0000 7faa0a7fc700 1 -- 172.21.15.190:0/1005384793 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faa10014420 con 0x7faa140f6510 2024-01-16T17:14:18.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.930+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa140fbf60 con 0x7faa140f6510 2024-01-16T17:14:18.910 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.931+0000 7faa0a7fc700 1 -- 172.21.15.190:0/1005384793 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faa10010230 con 0x7faa140f6510 2024-01-16T17:14:18.911 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:18.911 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:18.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faa0002ff60 msgr2=0x7faa00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 --2- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faa0002ff60 0x7faa00032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7faa040096a0 tx=0x7faa04006b40).stop 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 msgr2=0x7faa14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 --2- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 0x7faa14106730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7faa10013510 tx=0x7faa1000b040).stop 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 shutdown_connections 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 --2- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faa0002ff60 0x7faa00032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 --2- 172.21.15.190:0/1005384793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa140f6510 0x7faa14106730 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 >> 172.21.15.190:0/1005384793 conn(0x7faa140f1c70 msgr2=0x7faa140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 shutdown_connections 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:18.932+0000 7faa1bbac700 1 -- 172.21.15.190:0/1005384793 wait complete. 2024-01-16T17:14:18.913 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:19.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:17.815037+0000 mgr.smithi154.sllivz (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:19.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:18.931779+0000 mon.smithi154 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.190:0/1005384793' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:20.283 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:20.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.332+0000 7f2a34841700 1 -- 172.21.15.190:0/2619169574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 msgr2=0x7f2a300f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.332+0000 7f2a34841700 1 --2- 172.21.15.190:0/2619169574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 0x7f2a300f4c20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f2a24009fb0 tx=0x7f2a24009b30).stop 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.332+0000 7f2a34841700 1 -- 172.21.15.190:0/2619169574 shutdown_connections 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.332+0000 7f2a34841700 1 --2- 172.21.15.190:0/2619169574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 0x7f2a300f4c20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.332+0000 7f2a34841700 1 -- 172.21.15.190:0/2619169574 >> 172.21.15.190:0/2619169574 conn(0x7f2a300f0260 msgr2=0x7f2a300f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.332+0000 7f2a34841700 1 -- 172.21.15.190:0/2619169574 shutdown_connections 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.332+0000 7f2a34841700 1 -- 172.21.15.190:0/2619169574 wait complete. 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.332+0000 7f2a34841700 1 Processor -- start 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a34841700 1 -- start start 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a34841700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 0x7f2a301066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:21.314 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a34841700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a30106ba0 con 0x7f2a300f4800 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a2e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 0x7f2a301066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a2e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 0x7f2a301066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46932/0 (socket says 172.21.15.190:46932) 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a2e59c700 1 -- 172.21.15.190:0/622007272 learned_addr learned my addr 172.21.15.190:0/622007272 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a2e59c700 1 -- 172.21.15.190:0/622007272 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24005040 con 0x7f2a300f4800 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a2e59c700 1 --2- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 0x7f2a301066b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f2a24013510 tx=0x7f2a2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a1effd700 1 -- 172.21.15.190:0/622007272 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a24009320 con 0x7f2a300f4800 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a30106da0 con 0x7f2a300f4800 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a1effd700 1 -- 172.21.15.190:0/622007272 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2a24013f00 con 0x7f2a300f4800 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a1effd700 1 -- 172.21.15.190:0/622007272 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a24014dc0 con 0x7f2a300f4800 2024-01-16T17:14:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.333+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a30107230 con 0x7f2a300f4800 2024-01-16T17:14:21.316 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.334+0000 7f2a1effd700 1 -- 172.21.15.190:0/622007272 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2a240095f0 con 0x7f2a300f4800 2024-01-16T17:14:21.316 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.334+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a30045ad0 con 0x7f2a300f4800 2024-01-16T17:14:21.316 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.335+0000 7f2a1effd700 1 --2- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a1802ff60 0x7f2a18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:21.316 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.335+0000 7f2a1effd700 1 -- 172.21.15.190:0/622007272 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2a24041020 con 0x7f2a300f4800 2024-01-16T17:14:21.316 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.335+0000 7f2a2dd9b700 1 --2- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a1802ff60 0x7f2a18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:21.316 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.335+0000 7f2a2dd9b700 1 --2- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a1802ff60 0x7f2a18032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2a200096a0 tx=0x7f2a20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:21.319 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.339+0000 7f2a1effd700 1 -- 172.21.15.190:0/622007272 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2a24014420 con 0x7f2a300f4800 2024-01-16T17:14:21.531 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.552+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2a30108080 con 0x7f2a300f4800 2024-01-16T17:14:21.532 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.553+0000 7f2a1effd700 1 -- 172.21.15.190:0/622007272 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2a2401c920 con 0x7f2a300f4800 2024-01-16T17:14:21.533 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:21.533 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:21.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.554+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a1802ff60 msgr2=0x7f2a18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:21.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.554+0000 7f2a34841700 1 --2- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a1802ff60 0x7f2a18032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2a200096a0 tx=0x7f2a20006b40).stop 2024-01-16T17:14:21.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.554+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 msgr2=0x7f2a301066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.554+0000 7f2a34841700 1 --2- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 0x7f2a301066b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f2a24013510 tx=0x7f2a2400b040).stop 2024-01-16T17:14:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.554+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 shutdown_connections 2024-01-16T17:14:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.554+0000 7f2a34841700 1 --2- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2a1802ff60 0x7f2a18032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.554+0000 7f2a34841700 1 --2- 172.21.15.190:0/622007272 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a300f4800 0x7f2a301066b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.554+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 >> 172.21.15.190:0/622007272 conn(0x7f2a300f0260 msgr2=0x7f2a300f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.555+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 shutdown_connections 2024-01-16T17:14:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:21.555+0000 7f2a34841700 1 -- 172.21.15.190:0/622007272 wait complete. 2024-01-16T17:14:21.535 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:21.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:19.815331+0000 mgr.smithi154.sllivz (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:21.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:21.553449+0000 mon.smithi154 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.190:0/622007272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:22.911 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:22.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:23.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:21.815657+0000 mgr.smithi154.sllivz (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:24.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.027+0000 7fdd24a3f700 1 -- 172.21.15.190:0/1734303029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 msgr2=0x7fdd200f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:24.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.027+0000 7fdd24a3f700 1 --2- 172.21.15.190:0/1734303029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 0x7fdd200f6850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fdd10009fb0 tx=0x7fdd10009b30).stop 2024-01-16T17:14:24.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.027+0000 7fdd24a3f700 1 -- 172.21.15.190:0/1734303029 shutdown_connections 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.027+0000 7fdd24a3f700 1 --2- 172.21.15.190:0/1734303029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 0x7fdd200f6850 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.027+0000 7fdd24a3f700 1 -- 172.21.15.190:0/1734303029 >> 172.21.15.190:0/1734303029 conn(0x7fdd200f1b90 msgr2=0x7fdd200f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.027+0000 7fdd24a3f700 1 -- 172.21.15.190:0/1734303029 shutdown_connections 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.027+0000 7fdd24a3f700 1 -- 172.21.15.190:0/1734303029 wait complete. 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.028+0000 7fdd24a3f700 1 Processor -- start 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.028+0000 7fdd24a3f700 1 -- start start 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.028+0000 7fdd24a3f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 0x7fdd2006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.028+0000 7fdd24a3f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd20069e20 con 0x7fdd200f6430 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.028+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 0x7fdd2006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.028+0000 7fdd1e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 0x7fdd2006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:37096/0 (socket says 172.21.15.190:37096) 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.028+0000 7fdd1e59c700 1 -- 172.21.15.190:0/3601946891 learned_addr learned my addr 172.21.15.190:0/3601946891 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.028+0000 7fdd1e59c700 1 -- 172.21.15.190:0/3601946891 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd10005040 con 0x7fdd200f6430 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.029+0000 7fdd1e59c700 1 --2- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 0x7fdd2006b9a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fdd10013510 tx=0x7fdd1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.029+0000 7fdd0effd700 1 -- 172.21.15.190:0/3601946891 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd10009320 con 0x7fdd200f6430 2024-01-16T17:14:24.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.029+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd20069fc0 con 0x7fdd200f6430 2024-01-16T17:14:24.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.029+0000 7fdd0effd700 1 -- 172.21.15.190:0/3601946891 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd10013f00 con 0x7fdd200f6430 2024-01-16T17:14:24.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.029+0000 7fdd0effd700 1 -- 172.21.15.190:0/3601946891 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd10014d90 con 0x7fdd200f6430 2024-01-16T17:14:24.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.029+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2006a450 con 0x7fdd200f6430 2024-01-16T17:14:24.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.030+0000 7fdd0effd700 1 -- 172.21.15.190:0/3601946891 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd10014420 con 0x7fdd200f6430 2024-01-16T17:14:24.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.030+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd20045ad0 con 0x7fdd200f6430 2024-01-16T17:14:24.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.030+0000 7fdd0effd700 1 --2- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdd0802ff60 0x7fdd08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:24.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.030+0000 7fdd0effd700 1 -- 172.21.15.190:0/3601946891 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd10040fb0 con 0x7fdd200f6430 2024-01-16T17:14:24.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.030+0000 7fdd1dd9b700 1 --2- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdd0802ff60 0x7fdd08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:24.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.030+0000 7fdd1dd9b700 1 --2- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdd0802ff60 0x7fdd08032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdd140096a0 tx=0x7fdd14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:24.014 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.034+0000 7fdd0effd700 1 -- 172.21.15.190:0/3601946891 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd100095f0 con 0x7fdd200f6430 2024-01-16T17:14:24.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.240+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd2005ea50 con 0x7fdd200f6430 2024-01-16T17:14:24.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.241+0000 7fdd0effd700 1 -- 172.21.15.190:0/3601946891 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd1001db70 con 0x7fdd200f6430 2024-01-16T17:14:24.224 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:24.224 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:24.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdd0802ff60 msgr2=0x7fdd08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 --2- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdd0802ff60 0x7fdd08032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fdd140096a0 tx=0x7fdd14006b40).stop 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 msgr2=0x7fdd2006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 --2- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 0x7fdd2006b9a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fdd10013510 tx=0x7fdd1000b040).stop 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 shutdown_connections 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 --2- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdd0802ff60 0x7fdd08032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 --2- 172.21.15.190:0/3601946891 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdd200f6430 0x7fdd2006b9a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 >> 172.21.15.190:0/3601946891 conn(0x7fdd200f1b90 msgr2=0x7fdd200f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 shutdown_connections 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:24.243+0000 7fdd24a3f700 1 -- 172.21.15.190:0/3601946891 wait complete. 2024-01-16T17:14:24.225 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:24.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:24 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:24.241850+0000 mon.smithi154 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.190:0/3601946891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:25.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:25.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:25.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:23.815939+0000 mgr.smithi154.sllivz (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:26.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.747+0000 7f44589f3700 1 -- 172.21.15.190:0/2304241821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 msgr2=0x7f44540f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:26.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.747+0000 7f44589f3700 1 --2- 172.21.15.190:0/2304241821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 0x7f44540f9150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f4448009fb0 tx=0x7f4448009b30).stop 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.747+0000 7f44589f3700 1 -- 172.21.15.190:0/2304241821 shutdown_connections 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.747+0000 7f44589f3700 1 --2- 172.21.15.190:0/2304241821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 0x7f44540f9150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.747+0000 7f44589f3700 1 -- 172.21.15.190:0/2304241821 >> 172.21.15.190:0/2304241821 conn(0x7f44540f0b70 msgr2=0x7f44540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.747+0000 7f44589f3700 1 -- 172.21.15.190:0/2304241821 shutdown_connections 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.747+0000 7f44589f3700 1 -- 172.21.15.190:0/2304241821 wait complete. 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.747+0000 7f44589f3700 1 Processor -- start 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f44589f3700 1 -- start start 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f44589f3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 0x7f44541066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f44589f3700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4454106bd0 con 0x7f4454060220 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f445259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 0x7f44541066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f445259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 0x7f44541066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:37122/0 (socket says 172.21.15.190:37122) 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f445259c700 1 -- 172.21.15.190:0/3099498641 learned_addr learned my addr 172.21.15.190:0/3099498641 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f445259c700 1 -- 172.21.15.190:0/3099498641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4448005040 con 0x7f4454060220 2024-01-16T17:14:26.730 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f445259c700 1 --2- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 0x7f44541066e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f4448013510 tx=0x7f444800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:26.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f4442ffd700 1 -- 172.21.15.190:0/3099498641 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4448009320 con 0x7f4454060220 2024-01-16T17:14:26.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f4442ffd700 1 -- 172.21.15.190:0/3099498641 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4448013f00 con 0x7f4454060220 2024-01-16T17:14:26.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4454106dd0 con 0x7f4454060220 2024-01-16T17:14:26.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f4442ffd700 1 -- 172.21.15.190:0/3099498641 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4448014dc0 con 0x7f4454060220 2024-01-16T17:14:26.731 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.748+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4454107260 con 0x7f4454060220 2024-01-16T17:14:26.732 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.749+0000 7f4442ffd700 1 -- 172.21.15.190:0/3099498641 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44480095f0 con 0x7f4454060220 2024-01-16T17:14:26.732 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.749+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4454045ad0 con 0x7f4454060220 2024-01-16T17:14:26.732 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.749+0000 7f4442ffd700 1 --2- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f443c02ff60 0x7f443c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:26.732 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.750+0000 7f4442ffd700 1 -- 172.21.15.190:0/3099498641 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4448041040 con 0x7f4454060220 2024-01-16T17:14:26.732 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.750+0000 7f4451d9b700 1 --2- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f443c02ff60 0x7f443c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:26.732 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.750+0000 7f4451d9b700 1 --2- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f443c02ff60 0x7f443c032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f44440096a0 tx=0x7f4444006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:26.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.754+0000 7f4442ffd700 1 -- 172.21.15.190:0/3099498641 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4448014420 con 0x7f4454060220 2024-01-16T17:14:26.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.958+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4454108110 con 0x7f4454060220 2024-01-16T17:14:26.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.959+0000 7f4442ffd700 1 -- 172.21.15.190:0/3099498641 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4448010230 con 0x7f4454060220 2024-01-16T17:14:26.940 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:26.940 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:26.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.960+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f443c02ff60 msgr2=0x7f443c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.960+0000 7f44589f3700 1 --2- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f443c02ff60 0x7f443c032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f44440096a0 tx=0x7f4444006b40).stop 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.960+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 msgr2=0x7f44541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.960+0000 7f44589f3700 1 --2- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 0x7f44541066e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f4448013510 tx=0x7f444800b040).stop 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.961+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 shutdown_connections 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.961+0000 7f44589f3700 1 --2- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f443c02ff60 0x7f443c032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.961+0000 7f44589f3700 1 --2- 172.21.15.190:0/3099498641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4454060220 0x7f44541066e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.961+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 >> 172.21.15.190:0/3099498641 conn(0x7f44540f0b70 msgr2=0x7f44540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.961+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 shutdown_connections 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:26.961+0000 7f44589f3700 1 -- 172.21.15.190:0/3099498641 wait complete. 2024-01-16T17:14:26.942 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:27.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:25.816272+0000 mgr.smithi154.sllivz (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:27.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:26.960763+0000 mon.smithi154 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.190:0/3099498641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:28.276 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:28.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.401+0000 7ffa00a9a700 1 -- 172.21.15.190:0/1549076503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 msgr2=0x7ff9fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.401+0000 7ffa00a9a700 1 --2- 172.21.15.190:0/1549076503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 0x7ff9fc0f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7ff9e4009fb0 tx=0x7ff9e4009b30).stop 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.401+0000 7ffa00a9a700 1 -- 172.21.15.190:0/1549076503 shutdown_connections 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.401+0000 7ffa00a9a700 1 --2- 172.21.15.190:0/1549076503 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 0x7ff9fc0f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.401+0000 7ffa00a9a700 1 -- 172.21.15.190:0/1549076503 >> 172.21.15.190:0/1549076503 conn(0x7ff9fc0f0b70 msgr2=0x7ff9fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.401+0000 7ffa00a9a700 1 -- 172.21.15.190:0/1549076503 shutdown_connections 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.402+0000 7ffa00a9a700 1 -- 172.21.15.190:0/1549076503 wait complete. 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.402+0000 7ffa00a9a700 1 Processor -- start 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.402+0000 7ffa00a9a700 1 -- start start 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.402+0000 7ffa00a9a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 0x7ff9fc106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:29.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.402+0000 7ffa00a9a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc106c10 con 0x7ff9fc0f5410 2024-01-16T17:14:29.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.402+0000 7ff9fa59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 0x7ff9fc106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:29.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ff9fa59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 0x7ff9fc106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:37142/0 (socket says 172.21.15.190:37142) 2024-01-16T17:14:29.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ff9fa59c700 1 -- 172.21.15.190:0/4247463664 learned_addr learned my addr 172.21.15.190:0/4247463664 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:29.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ff9fa59c700 1 -- 172.21.15.190:0/4247463664 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e4005040 con 0x7ff9fc0f5410 2024-01-16T17:14:29.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ff9fa59c700 1 --2- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 0x7ff9fc106720 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff9e4014000 tx=0x7ff9e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:29.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ff9f2ffd700 1 -- 172.21.15.190:0/4247463664 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e40095f0 con 0x7ff9fc0f5410 2024-01-16T17:14:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9fc106e10 con 0x7ff9fc0f5410 2024-01-16T17:14:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ff9f2ffd700 1 -- 172.21.15.190:0/4247463664 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff9e4007e70 con 0x7ff9fc0f5410 2024-01-16T17:14:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ff9f2ffd700 1 -- 172.21.15.190:0/4247463664 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9e4014ea0 con 0x7ff9fc0f5410 2024-01-16T17:14:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.403+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9fc1072a0 con 0x7ff9fc0f5410 2024-01-16T17:14:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.404+0000 7ff9f2ffd700 1 -- 172.21.15.190:0/4247463664 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9e401d4e0 con 0x7ff9fc0f5410 2024-01-16T17:14:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.404+0000 7ff9f2ffd700 1 --2- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff9e802ff60 0x7ff9e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:29.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.404+0000 7ff9f2ffd700 1 -- 172.21.15.190:0/4247463664 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff9e4040930 con 0x7ff9fc0f5410 2024-01-16T17:14:29.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.404+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9fc045ad0 con 0x7ff9fc0f5410 2024-01-16T17:14:29.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.404+0000 7ff9f9d9b700 1 --2- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff9e802ff60 0x7ff9e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:29.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.405+0000 7ff9f9d9b700 1 --2- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff9e802ff60 0x7ff9e8032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff9ec0096a0 tx=0x7ff9ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:29.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.409+0000 7ff9f2ffd700 1 -- 172.21.15.190:0/4247463664 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff9e400f8f0 con 0x7ff9fc0f5410 2024-01-16T17:14:29.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.617+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9fc107f20 con 0x7ff9fc0f5410 2024-01-16T17:14:29.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.618+0000 7ff9f2ffd700 1 -- 172.21.15.190:0/4247463664 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff9e402c090 con 0x7ff9fc0f5410 2024-01-16T17:14:29.599 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:29.599 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:29.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff9e802ff60 msgr2=0x7ff9e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:29.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 --2- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff9e802ff60 0x7ff9e8032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff9ec0096a0 tx=0x7ff9ec006b40).stop 2024-01-16T17:14:29.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 msgr2=0x7ff9fc106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:29.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 --2- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 0x7ff9fc106720 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff9e4014000 tx=0x7ff9e400b040).stop 2024-01-16T17:14:29.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 shutdown_connections 2024-01-16T17:14:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 --2- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff9e802ff60 0x7ff9e8032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 --2- 172.21.15.190:0/4247463664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9fc0f5410 0x7ff9fc106720 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 >> 172.21.15.190:0/4247463664 conn(0x7ff9fc0f0b70 msgr2=0x7ff9fc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 shutdown_connections 2024-01-16T17:14:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:29.619+0000 7ffa00a9a700 1 -- 172.21.15.190:0/4247463664 wait complete. 2024-01-16T17:14:29.601 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:29.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:27.816586+0000 mgr.smithi154.sllivz (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:29.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:29.619603+0000 mon.smithi154 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.190:0/4247463664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:30.990 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:30.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:31.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:29.816820+0000 mgr.smithi154.sllivz (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:32.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.142+0000 7fd9041a7700 1 -- 172.21.15.190:0/3382025829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 msgr2=0x7fd8fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:32.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.142+0000 7fd9041a7700 1 --2- 172.21.15.190:0/3382025829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 0x7fd8fc0f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fd8f8009fb0 tx=0x7fd8f8009b30).stop 2024-01-16T17:14:32.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.142+0000 7fd9041a7700 1 -- 172.21.15.190:0/3382025829 shutdown_connections 2024-01-16T17:14:32.123 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.142+0000 7fd9041a7700 1 --2- 172.21.15.190:0/3382025829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 0x7fd8fc0f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:32.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.142+0000 7fd9041a7700 1 -- 172.21.15.190:0/3382025829 >> 172.21.15.190:0/3382025829 conn(0x7fd8fc0f0b70 msgr2=0x7fd8fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:32.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.142+0000 7fd9041a7700 1 -- 172.21.15.190:0/3382025829 shutdown_connections 2024-01-16T17:14:32.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.142+0000 7fd9041a7700 1 -- 172.21.15.190:0/3382025829 wait complete. 2024-01-16T17:14:32.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd9041a7700 1 Processor -- start 2024-01-16T17:14:32.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd9041a7700 1 -- start start 2024-01-16T17:14:32.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd9041a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 0x7fd8fc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:32.124 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd9041a7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc105af0 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd901f43700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 0x7fd8fc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd901f43700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 0x7fd8fc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:37162/0 (socket says 172.21.15.190:37162) 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd901f43700 1 -- 172.21.15.190:0/2522498105 learned_addr learned my addr 172.21.15.190:0/2522498105 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd901f43700 1 -- 172.21.15.190:0/2522498105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f8005040 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.143+0000 7fd901f43700 1 --2- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 0x7fd8fc105600 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd8f8014000 tx=0x7fd8f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.144+0000 7fd8ee7fc700 1 -- 172.21.15.190:0/2522498105 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f80095f0 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.144+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8fc105cf0 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.144+0000 7fd8ee7fc700 1 -- 172.21.15.190:0/2522498105 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8f8007e70 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.144+0000 7fd8ee7fc700 1 -- 172.21.15.190:0/2522498105 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f8014ea0 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.144+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc106180 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.144+0000 7fd8ee7fc700 1 -- 172.21.15.190:0/2522498105 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8f8025070 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.144+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc045ad0 con 0x7fd8fc0f5410 2024-01-16T17:14:32.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.145+0000 7fd8ee7fc700 1 --2- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd8e802ff60 0x7fd8e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:32.127 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.145+0000 7fd8ee7fc700 1 -- 172.21.15.190:0/2522498105 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8f8040a30 con 0x7fd8fc0f5410 2024-01-16T17:14:32.127 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.145+0000 7fd901742700 1 --2- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd8e802ff60 0x7fd8e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:32.127 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.145+0000 7fd901742700 1 --2- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd8e802ff60 0x7fd8e8032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd8f00096a0 tx=0x7fd8f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:32.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.149+0000 7fd8ee7fc700 1 -- 172.21.15.190:0/2522498105 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8f80079c0 con 0x7fd8fc0f5410 2024-01-16T17:14:32.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.354+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8fc0fae60 con 0x7fd8fc0f5410 2024-01-16T17:14:32.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.355+0000 7fd8ee7fc700 1 -- 172.21.15.190:0/2522498105 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd8f800f300 con 0x7fd8fc0f5410 2024-01-16T17:14:32.336 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:32.336 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd8e802ff60 msgr2=0x7fd8e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 --2- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd8e802ff60 0x7fd8e8032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fd8f00096a0 tx=0x7fd8f0006b40).stop 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 msgr2=0x7fd8fc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 --2- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 0x7fd8fc105600 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fd8f8014000 tx=0x7fd8f800b040).stop 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 shutdown_connections 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 --2- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd8e802ff60 0x7fd8e8032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 --2- 172.21.15.190:0/2522498105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8fc0f5410 0x7fd8fc105600 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 >> 172.21.15.190:0/2522498105 conn(0x7fd8fc0f0b70 msgr2=0x7fd8fc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 shutdown_connections 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:32.357+0000 7fd9041a7700 1 -- 172.21.15.190:0/2522498105 wait complete. 2024-01-16T17:14:32.338 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:32.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:32 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:32.357142+0000 mon.smithi154 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.190:0/2522498105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:33.728 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:33.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:33.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:31.817122+0000 mgr.smithi154.sllivz (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:34.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.896+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/760881995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 msgr2=0x7f8f540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:34.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.896+0000 7f8f5a2e5700 1 --2- 172.21.15.190:0/760881995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 0x7f8f540f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f8f44009fb0 tx=0x7f8f44009b30).stop 2024-01-16T17:14:34.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.896+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/760881995 shutdown_connections 2024-01-16T17:14:34.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.896+0000 7f8f5a2e5700 1 --2- 172.21.15.190:0/760881995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 0x7f8f540f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:34.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.896+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/760881995 >> 172.21.15.190:0/760881995 conn(0x7f8f540f0b70 msgr2=0x7f8f540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:34.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.896+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/760881995 shutdown_connections 2024-01-16T17:14:34.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.896+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/760881995 wait complete. 2024-01-16T17:14:34.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.897+0000 7f8f5a2e5700 1 Processor -- start 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.897+0000 7f8f5a2e5700 1 -- start start 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.897+0000 7f8f5a2e5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 0x7f8f541093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.897+0000 7f8f5a2e5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f54060280 con 0x7f8f540f5410 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f53fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 0x7f8f541093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f53fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 0x7f8f541093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60838/0 (socket says 172.21.15.190:60838) 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f53fff700 1 -- 172.21.15.190:0/2494314022 learned_addr learned my addr 172.21.15.190:0/2494314022 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f53fff700 1 -- 172.21.15.190:0/2494314022 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f44005040 con 0x7f8f540f5410 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f53fff700 1 --2- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 0x7f8f541093f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8f44013510 tx=0x7f8f4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f50ff9700 1 -- 172.21.15.190:0/2494314022 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f44009320 con 0x7f8f540f5410 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f54060480 con 0x7f8f540f5410 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f50ff9700 1 -- 172.21.15.190:0/2494314022 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f44013f00 con 0x7f8f540f5410 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f50ff9700 1 -- 172.21.15.190:0/2494314022 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f44014d90 con 0x7f8f540f5410 2024-01-16T17:14:34.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.898+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f54109b20 con 0x7f8f540f5410 2024-01-16T17:14:34.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.899+0000 7f8f50ff9700 1 -- 172.21.15.190:0/2494314022 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f440263a0 con 0x7f8f540f5410 2024-01-16T17:14:34.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.899+0000 7f8f50ff9700 1 --2- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3c02ff60 0x7f8f3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:34.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.899+0000 7f8f50ff9700 1 -- 172.21.15.190:0/2494314022 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f440410d0 con 0x7f8f540f5410 2024-01-16T17:14:34.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.899+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f54045ad0 con 0x7f8f540f5410 2024-01-16T17:14:34.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.899+0000 7f8f537fe700 1 --2- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3c02ff60 0x7f8f3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:34.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.900+0000 7f8f537fe700 1 --2- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3c02ff60 0x7f8f3c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8f480096a0 tx=0x7f8f48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:34.885 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:34.905+0000 7f8f50ff9700 1 -- 172.21.15.190:0/2494314022 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f44012920 con 0x7f8f540f5410 2024-01-16T17:14:35.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.110+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f5410a240 con 0x7f8f540f5410 2024-01-16T17:14:35.091 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.110+0000 7f8f50ff9700 1 -- 172.21.15.190:0/2494314022 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8f4402b020 con 0x7f8f540f5410 2024-01-16T17:14:35.091 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:35.091 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3c02ff60 msgr2=0x7f8f3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 --2- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3c02ff60 0x7f8f3c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f8f480096a0 tx=0x7f8f48006b40).stop 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 msgr2=0x7f8f541093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 --2- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 0x7f8f541093f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f8f44013510 tx=0x7f8f4400b040).stop 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 shutdown_connections 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 --2- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3c02ff60 0x7f8f3c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 --2- 172.21.15.190:0/2494314022 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f540f5410 0x7f8f541093f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 >> 172.21.15.190:0/2494314022 conn(0x7f8f540f0b70 msgr2=0x7f8f540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 shutdown_connections 2024-01-16T17:14:35.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:35.112+0000 7f8f5a2e5700 1 -- 172.21.15.190:0/2494314022 wait complete. 2024-01-16T17:14:35.094 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:35.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:33.817382+0000 mgr.smithi154.sllivz (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:35.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:35.112617+0000 mon.smithi154 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.190:0/2494314022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:36.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:36.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.553+0000 7f4227137700 1 -- 172.21.15.190:0/1250033184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 msgr2=0x7f42200f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.553+0000 7f4227137700 1 --2- 172.21.15.190:0/1250033184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 0x7f42200f4be0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f4214009fb0 tx=0x7f4214009b30).stop 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.553+0000 7f4227137700 1 -- 172.21.15.190:0/1250033184 shutdown_connections 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.553+0000 7f4227137700 1 --2- 172.21.15.190:0/1250033184 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 0x7f42200f4be0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.553+0000 7f4227137700 1 -- 172.21.15.190:0/1250033184 >> 172.21.15.190:0/1250033184 conn(0x7f42200f02e0 msgr2=0x7f42200f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.553+0000 7f4227137700 1 -- 172.21.15.190:0/1250033184 shutdown_connections 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.553+0000 7f4227137700 1 -- 172.21.15.190:0/1250033184 wait complete. 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.554+0000 7f4227137700 1 Processor -- start 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.554+0000 7f4227137700 1 -- start start 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.554+0000 7f4227137700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 0x7f4220106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:37.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.554+0000 7f4227137700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4220106bf0 con 0x7f42200f47c0 2024-01-16T17:14:37.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f4224ed3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 0x7f4220106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:37.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f4224ed3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 0x7f4220106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60858/0 (socket says 172.21.15.190:60858) 2024-01-16T17:14:37.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f4224ed3700 1 -- 172.21.15.190:0/1105773909 learned_addr learned my addr 172.21.15.190:0/1105773909 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:37.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f4224ed3700 1 -- 172.21.15.190:0/1105773909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4214005040 con 0x7f42200f47c0 2024-01-16T17:14:37.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f4224ed3700 1 --2- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 0x7f4220106700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4214013510 tx=0x7f421400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:37.536 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f421d7fa700 1 -- 172.21.15.190:0/1105773909 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4214009320 con 0x7f42200f47c0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4220106df0 con 0x7f42200f47c0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f421d7fa700 1 -- 172.21.15.190:0/1105773909 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4214013f00 con 0x7f42200f47c0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f421d7fa700 1 -- 172.21.15.190:0/1105773909 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4214014d90 con 0x7f42200f47c0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.555+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4220107280 con 0x7f42200f47c0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.556+0000 7f421d7fa700 1 -- 172.21.15.190:0/1105773909 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42140263a0 con 0x7f42200f47c0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.556+0000 7f421d7fa700 1 --2- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f421002ff60 0x7f4210032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.556+0000 7f421d7fa700 1 -- 172.21.15.190:0/1105773909 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42140410c0 con 0x7f42200f47c0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.556+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4220045ad0 con 0x7f42200f47c0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.556+0000 7f421ffff700 1 --2- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f421002ff60 0x7f4210032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:37.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.557+0000 7f421ffff700 1 --2- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f421002ff60 0x7f4210032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f420c0096a0 tx=0x7f420c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:37.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.561+0000 7f421d7fa700 1 -- 172.21.15.190:0/1105773909 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4214012920 con 0x7f42200f47c0 2024-01-16T17:14:37.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.768+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42200fc050 con 0x7f42200f47c0 2024-01-16T17:14:37.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.768+0000 7f421d7fa700 1 -- 172.21.15.190:0/1105773909 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f421402b020 con 0x7f42200f47c0 2024-01-16T17:14:37.748 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:37.748 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:37.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f421002ff60 msgr2=0x7f4210032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:37.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 --2- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f421002ff60 0x7f4210032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f420c0096a0 tx=0x7f420c006b40).stop 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 msgr2=0x7f4220106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 --2- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 0x7f4220106700 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f4214013510 tx=0x7f421400b040).stop 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 shutdown_connections 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 --2- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f421002ff60 0x7f4210032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 --2- 172.21.15.190:0/1105773909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42200f47c0 0x7f4220106700 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 >> 172.21.15.190:0/1105773909 conn(0x7f42200f02e0 msgr2=0x7f42200f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 shutdown_connections 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:37.770+0000 7f4227137700 1 -- 172.21.15.190:0/1105773909 wait complete. 2024-01-16T17:14:37.750 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:37.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:35.817702+0000 mgr.smithi154.sllivz (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:38.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:38 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:37.769284+0000 mon.smithi154 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.190:0/1105773909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:39.113 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:39.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:39.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:37.818062+0000 mgr.smithi154.sllivz (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 -- 172.21.15.190:0/2612764525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 msgr2=0x7ff9200f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 --2- 172.21.15.190:0/2612764525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 0x7ff9200f5d70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7ff910009fb0 tx=0x7ff910009b30).stop 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 -- 172.21.15.190:0/2612764525 shutdown_connections 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 --2- 172.21.15.190:0/2612764525 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 0x7ff9200f5d70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 -- 172.21.15.190:0/2612764525 >> 172.21.15.190:0/2612764525 conn(0x7ff9200f0260 msgr2=0x7ff9200f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 -- 172.21.15.190:0/2612764525 shutdown_connections 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 -- 172.21.15.190:0/2612764525 wait complete. 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 Processor -- start 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 -- start start 2024-01-16T17:14:40.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 0x7ff9200fd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.267+0000 7ff927e42700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9200fb800 con 0x7ff9200f5950 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff925bde700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 0x7ff9200fd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff925bde700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 0x7ff9200fd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60886/0 (socket says 172.21.15.190:60886) 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff925bde700 1 -- 172.21.15.190:0/2526138113 learned_addr learned my addr 172.21.15.190:0/2526138113 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff925bde700 1 -- 172.21.15.190:0/2526138113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff910005040 con 0x7ff9200f5950 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff925bde700 1 --2- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 0x7ff9200fd380 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7ff910013510 tx=0x7ff91000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff9167fc700 1 -- 172.21.15.190:0/2526138113 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff910009320 con 0x7ff9200f5950 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9200fb9a0 con 0x7ff9200f5950 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff9167fc700 1 -- 172.21.15.190:0/2526138113 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff910013f00 con 0x7ff9200f5950 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff9167fc700 1 -- 172.21.15.190:0/2526138113 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff910014d90 con 0x7ff9200f5950 2024-01-16T17:14:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.268+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9200fbe30 con 0x7ff9200f5950 2024-01-16T17:14:40.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.269+0000 7ff9167fc700 1 -- 172.21.15.190:0/2526138113 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff9100095f0 con 0x7ff9200f5950 2024-01-16T17:14:40.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.269+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff920059100 con 0x7ff9200f5950 2024-01-16T17:14:40.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.269+0000 7ff9167fc700 1 --2- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff90c02ffb0 0x7ff90c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:40.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.269+0000 7ff9167fc700 1 -- 172.21.15.190:0/2526138113 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff910041020 con 0x7ff9200f5950 2024-01-16T17:14:40.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.269+0000 7ff9253dd700 1 --2- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff90c02ffb0 0x7ff90c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:40.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.270+0000 7ff9253dd700 1 --2- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff90c02ffb0 0x7ff90c032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7ff91c0096a0 tx=0x7ff91c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:40.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.274+0000 7ff9167fc700 1 -- 172.21.15.190:0/2526138113 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff910014420 con 0x7ff9200f5950 2024-01-16T17:14:40.460 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.481+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff920045ad0 con 0x7ff9200f5950 2024-01-16T17:14:40.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.481+0000 7ff9167fc700 1 -- 172.21.15.190:0/2526138113 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff91001d7f0 con 0x7ff9200f5950 2024-01-16T17:14:40.461 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:40.461 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:40.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff90c02ffb0 msgr2=0x7ff90c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:40.463 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 --2- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff90c02ffb0 0x7ff90c032470 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7ff91c0096a0 tx=0x7ff91c006b40).stop 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 msgr2=0x7ff9200fd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 --2- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 0x7ff9200fd380 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7ff910013510 tx=0x7ff91000b040).stop 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 shutdown_connections 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 --2- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff90c02ffb0 0x7ff90c032470 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 --2- 172.21.15.190:0/2526138113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9200f5950 0x7ff9200fd380 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 >> 172.21.15.190:0/2526138113 conn(0x7ff9200f0260 msgr2=0x7ff9200f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 shutdown_connections 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:40.483+0000 7ff927e42700 1 -- 172.21.15.190:0/2526138113 wait complete. 2024-01-16T17:14:40.464 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:40.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:40 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:40.482439+0000 mon.smithi154 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.190:0/2526138113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:41.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:41.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:41.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:39.818404+0000 mgr.smithi154.sllivz (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:43.017 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.037+0000 7f6c32435700 1 -- 172.21.15.190:0/1728424722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 msgr2=0x7f6c2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.037+0000 7f6c32435700 1 --2- 172.21.15.190:0/1728424722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 0x7f6c2c0f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f6c20009fb0 tx=0x7f6c20009b30).stop 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.037+0000 7f6c32435700 1 -- 172.21.15.190:0/1728424722 shutdown_connections 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.037+0000 7f6c32435700 1 --2- 172.21.15.190:0/1728424722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 0x7f6c2c0f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.037+0000 7f6c32435700 1 -- 172.21.15.190:0/1728424722 >> 172.21.15.190:0/1728424722 conn(0x7f6c2c0f0b70 msgr2=0x7f6c2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.037+0000 7f6c32435700 1 -- 172.21.15.190:0/1728424722 shutdown_connections 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.037+0000 7f6c32435700 1 -- 172.21.15.190:0/1728424722 wait complete. 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.037+0000 7f6c32435700 1 Processor -- start 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.038+0000 7f6c32435700 1 -- start start 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.038+0000 7f6c32435700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 0x7f6c2c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:43.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.038+0000 7f6c32435700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c060220 con 0x7f6c2c0f5410 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.038+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 0x7f6c2c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.038+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 0x7f6c2c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48572/0 (socket says 172.21.15.190:48572) 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.038+0000 7f6c2bfff700 1 -- 172.21.15.190:0/1042530140 learned_addr learned my addr 172.21.15.190:0/1042530140 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.039+0000 7f6c2bfff700 1 -- 172.21.15.190:0/1042530140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c20005040 con 0x7f6c2c0f5410 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.039+0000 7f6c2bfff700 1 --2- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 0x7f6c2c109450 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6c20013510 tx=0x7f6c2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.039+0000 7f6c28ff9700 1 -- 172.21.15.190:0/1042530140 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c20009320 con 0x7f6c2c0f5410 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.039+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2c060420 con 0x7f6c2c0f5410 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.039+0000 7f6c28ff9700 1 -- 172.21.15.190:0/1042530140 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c20013f00 con 0x7f6c2c0f5410 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.039+0000 7f6c28ff9700 1 -- 172.21.15.190:0/1042530140 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c20014d90 con 0x7f6c2c0f5410 2024-01-16T17:14:43.020 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.039+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2c109b00 con 0x7f6c2c0f5410 2024-01-16T17:14:43.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.040+0000 7f6c28ff9700 1 -- 172.21.15.190:0/1042530140 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c200263a0 con 0x7f6c2c0f5410 2024-01-16T17:14:43.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.040+0000 7f6c28ff9700 1 --2- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c1402ff60 0x7f6c14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:43.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.040+0000 7f6c28ff9700 1 -- 172.21.15.190:0/1042530140 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c20040fc0 con 0x7f6c2c0f5410 2024-01-16T17:14:43.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.040+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2c059100 con 0x7f6c2c0f5410 2024-01-16T17:14:43.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.040+0000 7f6c2b7fe700 1 --2- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c1402ff60 0x7f6c14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:43.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.040+0000 7f6c2b7fe700 1 --2- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c1402ff60 0x7f6c14032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6c1c0096a0 tx=0x7f6c1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:43.024 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.044+0000 7f6c28ff9700 1 -- 172.21.15.190:0/1042530140 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c200095f0 con 0x7f6c2c0f5410 2024-01-16T17:14:43.234 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.255+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c2c10a1b0 con 0x7f6c2c0f5410 2024-01-16T17:14:43.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.256+0000 7f6c28ff9700 1 -- 172.21.15.190:0/1042530140 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6c2001db70 con 0x7f6c2c0f5410 2024-01-16T17:14:43.235 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:43.236 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c1402ff60 msgr2=0x7f6c14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 --2- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c1402ff60 0x7f6c14032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f6c1c0096a0 tx=0x7f6c1c006b40).stop 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 msgr2=0x7f6c2c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 --2- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 0x7f6c2c109450 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f6c20013510 tx=0x7f6c2000b040).stop 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 shutdown_connections 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 --2- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c1402ff60 0x7f6c14032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 --2- 172.21.15.190:0/1042530140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c2c0f5410 0x7f6c2c109450 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 >> 172.21.15.190:0/1042530140 conn(0x7f6c2c0f0b70 msgr2=0x7f6c2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 shutdown_connections 2024-01-16T17:14:43.238 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:43.258+0000 7f6c32435700 1 -- 172.21.15.190:0/1042530140 wait complete. 2024-01-16T17:14:43.239 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:43.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:41.818728+0000 mgr.smithi154.sllivz (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:43.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:43.256735+0000 mon.smithi154 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.190:0/1042530140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:44.612 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:44.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:45.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.766+0000 7f5ae4658700 1 -- 172.21.15.190:0/34371522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 msgr2=0x7f5adc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:45.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.766+0000 7f5ae4658700 1 --2- 172.21.15.190:0/34371522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 0x7f5adc0f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f5acc009fb0 tx=0x7f5acc009b30).stop 2024-01-16T17:14:45.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.767+0000 7f5ae4658700 1 -- 172.21.15.190:0/34371522 shutdown_connections 2024-01-16T17:14:45.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.767+0000 7f5ae4658700 1 --2- 172.21.15.190:0/34371522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 0x7f5adc0f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:45.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.767+0000 7f5ae4658700 1 -- 172.21.15.190:0/34371522 >> 172.21.15.190:0/34371522 conn(0x7f5adc0f0b70 msgr2=0x7f5adc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:45.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.767+0000 7f5ae4658700 1 -- 172.21.15.190:0/34371522 shutdown_connections 2024-01-16T17:14:45.747 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.767+0000 7f5ae4658700 1 -- 172.21.15.190:0/34371522 wait complete. 2024-01-16T17:14:45.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.767+0000 7f5ae4658700 1 Processor -- start 2024-01-16T17:14:45.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.767+0000 7f5ae4658700 1 -- start start 2024-01-16T17:14:45.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ae4658700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 0x7f5adc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:45.748 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ae4658700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc105b20 con 0x7f5adc0f5410 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ae23f4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 0x7f5adc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ae23f4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 0x7f5adc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48584/0 (socket says 172.21.15.190:48584) 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ae23f4700 1 -- 172.21.15.190:0/808890872 learned_addr learned my addr 172.21.15.190:0/808890872 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ae23f4700 1 -- 172.21.15.190:0/808890872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc005040 con 0x7f5adc0f5410 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ae23f4700 1 --2- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 0x7f5adc105630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5acc013510 tx=0x7f5acc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ad2ffd700 1 -- 172.21.15.190:0/808890872 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5acc009320 con 0x7f5adc0f5410 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc105d20 con 0x7f5adc0f5410 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ad2ffd700 1 -- 172.21.15.190:0/808890872 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5acc013f00 con 0x7f5adc0f5410 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.768+0000 7f5ad2ffd700 1 -- 172.21.15.190:0/808890872 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5acc014d90 con 0x7f5adc0f5410 2024-01-16T17:14:45.749 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.769+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc1061b0 con 0x7f5adc0f5410 2024-01-16T17:14:45.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.769+0000 7f5ad2ffd700 1 -- 172.21.15.190:0/808890872 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5acc0095f0 con 0x7f5adc0f5410 2024-01-16T17:14:45.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.769+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc045ad0 con 0x7f5adc0f5410 2024-01-16T17:14:45.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.769+0000 7f5ad2ffd700 1 --2- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ac802ff60 0x7f5ac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:45.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.769+0000 7f5ad2ffd700 1 -- 172.21.15.190:0/808890872 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5acc040fe0 con 0x7f5adc0f5410 2024-01-16T17:14:45.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.770+0000 7f5ae1bf3700 1 --2- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ac802ff60 0x7f5ac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:45.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.770+0000 7f5ae1bf3700 1 --2- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ac802ff60 0x7f5ac8032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5ad80096a0 tx=0x7f5ad8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:45.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.774+0000 7f5ad2ffd700 1 -- 172.21.15.190:0/808890872 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5acc01d7f0 con 0x7f5adc0f5410 2024-01-16T17:14:45.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:43.819018+0000 mgr.smithi154.sllivz (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:45.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.980+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5adc0fae60 con 0x7f5adc0f5410 2024-01-16T17:14:45.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.980+0000 7f5ad2ffd700 1 -- 172.21.15.190:0/808890872 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5acc010230 con 0x7f5adc0f5410 2024-01-16T17:14:45.960 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:45.961 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ac802ff60 msgr2=0x7f5ac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 --2- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ac802ff60 0x7f5ac8032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f5ad80096a0 tx=0x7f5ad8006b40).stop 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 msgr2=0x7f5adc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 --2- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 0x7f5adc105630 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f5acc013510 tx=0x7f5acc00b040).stop 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 shutdown_connections 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 --2- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ac802ff60 0x7f5ac8032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 --2- 172.21.15.190:0/808890872 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5adc0f5410 0x7f5adc105630 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 >> 172.21.15.190:0/808890872 conn(0x7f5adc0f0b70 msgr2=0x7f5adc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 shutdown_connections 2024-01-16T17:14:45.962 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:45.982+0000 7f5ae4658700 1 -- 172.21.15.190:0/808890872 wait complete. 2024-01-16T17:14:45.963 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:46.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:46 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:45.981326+0000 mon.smithi154 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.190:0/808890872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:47.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:47.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:47.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:45.819349+0000 mgr.smithi154.sllivz (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:48.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.492+0000 7fb4a8d31700 1 -- 172.21.15.190:0/3950947725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 msgr2=0x7fb4a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:48.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.492+0000 7fb4a8d31700 1 --2- 172.21.15.190:0/3950947725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 0x7fb4a40f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fb498009fb0 tx=0x7fb498009b30).stop 2024-01-16T17:14:48.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.492+0000 7fb4a8d31700 1 -- 172.21.15.190:0/3950947725 shutdown_connections 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.492+0000 7fb4a8d31700 1 --2- 172.21.15.190:0/3950947725 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 0x7fb4a40f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.492+0000 7fb4a8d31700 1 -- 172.21.15.190:0/3950947725 >> 172.21.15.190:0/3950947725 conn(0x7fb4a40f0b70 msgr2=0x7fb4a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.492+0000 7fb4a8d31700 1 -- 172.21.15.190:0/3950947725 shutdown_connections 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.492+0000 7fb4a8d31700 1 -- 172.21.15.190:0/3950947725 wait complete. 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.493+0000 7fb4a8d31700 1 Processor -- start 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.493+0000 7fb4a8d31700 1 -- start start 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.493+0000 7fb4a8d31700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 0x7fb4a4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.493+0000 7fb4a8d31700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4060220 con 0x7fb4a40f5410 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.493+0000 7fb4a259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 0x7fb4a4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.493+0000 7fb4a259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 0x7fb4a4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48602/0 (socket says 172.21.15.190:48602) 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.493+0000 7fb4a259c700 1 -- 172.21.15.190:0/1124538347 learned_addr learned my addr 172.21.15.190:0/1124538347 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.494+0000 7fb4a259c700 1 -- 172.21.15.190:0/1124538347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb498005040 con 0x7fb4a40f5410 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.494+0000 7fb4a259c700 1 --2- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 0x7fb4a4109450 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb498014000 tx=0x7fb49800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.494+0000 7fb492ffd700 1 -- 172.21.15.190:0/1124538347 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4980095f0 con 0x7fb4a40f5410 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.494+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a4060420 con 0x7fb4a40f5410 2024-01-16T17:14:48.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.494+0000 7fb492ffd700 1 -- 172.21.15.190:0/1124538347 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb498007e70 con 0x7fb4a40f5410 2024-01-16T17:14:48.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.494+0000 7fb492ffd700 1 -- 172.21.15.190:0/1124538347 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb498014ea0 con 0x7fb4a40f5410 2024-01-16T17:14:48.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.494+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a4109b00 con 0x7fb4a40f5410 2024-01-16T17:14:48.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.495+0000 7fb492ffd700 1 -- 172.21.15.190:0/1124538347 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb49801d4e0 con 0x7fb4a40f5410 2024-01-16T17:14:48.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.495+0000 7fb492ffd700 1 --2- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb48c02ff60 0x7fb48c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:48.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.495+0000 7fb492ffd700 1 -- 172.21.15.190:0/1124538347 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4980408a0 con 0x7fb4a40f5410 2024-01-16T17:14:48.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.495+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a4059100 con 0x7fb4a40f5410 2024-01-16T17:14:48.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.495+0000 7fb4a1d9b700 1 --2- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb48c02ff60 0x7fb48c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:48.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.495+0000 7fb4a1d9b700 1 --2- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb48c02ff60 0x7fb48c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb4940096a0 tx=0x7fb494006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:48.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.500+0000 7fb492ffd700 1 -- 172.21.15.190:0/1124538347 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb49800f8f0 con 0x7fb4a40f5410 2024-01-16T17:14:48.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.705+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4a410a1b0 con 0x7fb4a40f5410 2024-01-16T17:14:48.685 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.706+0000 7fb492ffd700 1 -- 172.21.15.190:0/1124538347 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb49802c090 con 0x7fb4a40f5410 2024-01-16T17:14:48.685 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:48.686 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb48c02ff60 msgr2=0x7fb48c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 --2- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb48c02ff60 0x7fb48c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb4940096a0 tx=0x7fb494006b40).stop 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 msgr2=0x7fb4a4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 --2- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 0x7fb4a4109450 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fb498014000 tx=0x7fb49800b040).stop 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 shutdown_connections 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 --2- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb48c02ff60 0x7fb48c032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 --2- 172.21.15.190:0/1124538347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4a40f5410 0x7fb4a4109450 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 >> 172.21.15.190:0/1124538347 conn(0x7fb4a40f0b70 msgr2=0x7fb4a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 shutdown_connections 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:48.708+0000 7fb4a8d31700 1 -- 172.21.15.190:0/1124538347 wait complete. 2024-01-16T17:14:48.688 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:49.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:47.819717+0000 mgr.smithi154.sllivz (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:49.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:48.706900+0000 mon.smithi154 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.190:0/1124538347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:50.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:50.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:51.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.224+0000 7f5f721e2700 1 -- 172.21.15.190:0/577822033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 msgr2=0x7f5f6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:51.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.224+0000 7f5f721e2700 1 --2- 172.21.15.190:0/577822033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 0x7f5f6c0f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f5f54009fb0 tx=0x7f5f54009b30).stop 2024-01-16T17:14:51.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.224+0000 7f5f721e2700 1 -- 172.21.15.190:0/577822033 shutdown_connections 2024-01-16T17:14:51.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.224+0000 7f5f721e2700 1 --2- 172.21.15.190:0/577822033 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 0x7f5f6c0f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:51.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.224+0000 7f5f721e2700 1 -- 172.21.15.190:0/577822033 >> 172.21.15.190:0/577822033 conn(0x7f5f6c0f1c70 msgr2=0x7f5f6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.224+0000 7f5f721e2700 1 -- 172.21.15.190:0/577822033 shutdown_connections 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.224+0000 7f5f721e2700 1 -- 172.21.15.190:0/577822033 wait complete. 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.224+0000 7f5f721e2700 1 Processor -- start 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f721e2700 1 -- start start 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f721e2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 0x7f5f6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f721e2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c106c20 con 0x7f5f6c0f6510 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f6b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 0x7f5f6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f6b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 0x7f5f6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48630/0 (socket says 172.21.15.190:48630) 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f6b7fe700 1 -- 172.21.15.190:0/1799332770 learned_addr learned my addr 172.21.15.190:0/1799332770 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f6b7fe700 1 -- 172.21.15.190:0/1799332770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f54005040 con 0x7f5f6c0f6510 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f6b7fe700 1 --2- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 0x7f5f6c106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f5f54013510 tx=0x7f5f5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f63fff700 1 -- 172.21.15.190:0/1799332770 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f54009320 con 0x7f5f6c0f6510 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f63fff700 1 -- 172.21.15.190:0/1799332770 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f54013f00 con 0x7f5f6c0f6510 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6c106e20 con 0x7f5f6c0f6510 2024-01-16T17:14:51.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f63fff700 1 -- 172.21.15.190:0/1799332770 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f54014d90 con 0x7f5f6c0f6510 2024-01-16T17:14:51.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.225+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6c1072b0 con 0x7f5f6c0f6510 2024-01-16T17:14:51.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.226+0000 7f5f63fff700 1 -- 172.21.15.190:0/1799332770 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f540095f0 con 0x7f5f6c0f6510 2024-01-16T17:14:51.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.226+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6c045ad0 con 0x7f5f6c0f6510 2024-01-16T17:14:51.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.226+0000 7f5f63fff700 1 --2- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f5802ff60 0x7f5f58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:51.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.226+0000 7f5f63fff700 1 -- 172.21.15.190:0/1799332770 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f54041000 con 0x7f5f6c0f6510 2024-01-16T17:14:51.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.227+0000 7f5f6affd700 1 --2- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f5802ff60 0x7f5f58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:51.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.227+0000 7f5f6affd700 1 --2- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f5802ff60 0x7f5f58032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5f5c0096a0 tx=0x7f5f5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:51.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.231+0000 7f5f63fff700 1 -- 172.21.15.190:0/1799332770 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f54014420 con 0x7f5f6c0f6510 2024-01-16T17:14:51.419 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.440+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f6c0fbf60 con 0x7f5f6c0f6510 2024-01-16T17:14:51.420 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.440+0000 7f5f63fff700 1 -- 172.21.15.190:0/1799332770 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f5401d7f0 con 0x7f5f6c0f6510 2024-01-16T17:14:51.420 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:51.420 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f5802ff60 msgr2=0x7f5f58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 --2- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f5802ff60 0x7f5f58032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f5f5c0096a0 tx=0x7f5f5c006b40).stop 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 msgr2=0x7f5f6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 --2- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 0x7f5f6c106730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f5f54013510 tx=0x7f5f5400b040).stop 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 shutdown_connections 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 --2- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f5802ff60 0x7f5f58032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 --2- 172.21.15.190:0/1799332770 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c0f6510 0x7f5f6c106730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 >> 172.21.15.190:0/1799332770 conn(0x7f5f6c0f1c70 msgr2=0x7f5f6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 shutdown_connections 2024-01-16T17:14:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:51.442+0000 7f5f721e2700 1 -- 172.21.15.190:0/1799332770 wait complete. 2024-01-16T17:14:51.423 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:51.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:49.820086+0000 mgr.smithi154.sllivz (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:51.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:51.441629+0000 mon.smithi154 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.190:0/1799332770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:52.842 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:52.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:53.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.967+0000 7f4396d6a700 1 -- 172.21.15.190:0/1901077527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 msgr2=0x7f43900f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.967+0000 7f4396d6a700 1 --2- 172.21.15.190:0/1901077527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 0x7f43900f90f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f4384009fb0 tx=0x7f4384009b30).stop 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.967+0000 7f4396d6a700 1 -- 172.21.15.190:0/1901077527 shutdown_connections 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.967+0000 7f4396d6a700 1 --2- 172.21.15.190:0/1901077527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 0x7f43900f90f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.967+0000 7f4396d6a700 1 -- 172.21.15.190:0/1901077527 >> 172.21.15.190:0/1901077527 conn(0x7f43900f0b70 msgr2=0x7f43900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.967+0000 7f4396d6a700 1 -- 172.21.15.190:0/1901077527 shutdown_connections 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.967+0000 7f4396d6a700 1 -- 172.21.15.190:0/1901077527 wait complete. 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4396d6a700 1 Processor -- start 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4396d6a700 1 -- start start 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4396d6a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 0x7f43901066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4396d6a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4390106bd0 con 0x7f4390060280 2024-01-16T17:14:53.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4394b06700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 0x7f43901066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4394b06700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 0x7f43901066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50224/0 (socket says 172.21.15.190:50224) 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4394b06700 1 -- 172.21.15.190:0/3085457004 learned_addr learned my addr 172.21.15.190:0/3085457004 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4394b06700 1 -- 172.21.15.190:0/3085457004 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4384005040 con 0x7f4390060280 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.968+0000 7f4394b06700 1 --2- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 0x7f43901066e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f4384013510 tx=0x7f438400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.969+0000 7f438d7fa700 1 -- 172.21.15.190:0/3085457004 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4384009320 con 0x7f4390060280 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.969+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4390106dd0 con 0x7f4390060280 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.969+0000 7f438d7fa700 1 -- 172.21.15.190:0/3085457004 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4384013f00 con 0x7f4390060280 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.969+0000 7f438d7fa700 1 -- 172.21.15.190:0/3085457004 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4384014dc0 con 0x7f4390060280 2024-01-16T17:14:53.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.969+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4390107260 con 0x7f4390060280 2024-01-16T17:14:53.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.969+0000 7f438d7fa700 1 -- 172.21.15.190:0/3085457004 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4384014420 con 0x7f4390060280 2024-01-16T17:14:53.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.970+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4390045ad0 con 0x7f4390060280 2024-01-16T17:14:53.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.970+0000 7f438d7fa700 1 --2- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f438002ff60 0x7f4380032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:53.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.970+0000 7f438d7fa700 1 -- 172.21.15.190:0/3085457004 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43840410f0 con 0x7f4390060280 2024-01-16T17:14:53.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.970+0000 7f438ffff700 1 --2- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f438002ff60 0x7f4380032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:53.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.970+0000 7f438ffff700 1 --2- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f438002ff60 0x7f4380032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f437c0096a0 tx=0x7f437c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:53.954 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:53.975+0000 7f438d7fa700 1 -- 172.21.15.190:0/3085457004 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43840095f0 con 0x7f4390060280 2024-01-16T17:14:53.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:51.820472+0000 mgr.smithi154.sllivz (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:54.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.182+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4390108110 con 0x7f4390060280 2024-01-16T17:14:54.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.183+0000 7f438d7fa700 1 -- 172.21.15.190:0/3085457004 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f438401cb10 con 0x7f4390060280 2024-01-16T17:14:54.162 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:54.162 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f438002ff60 msgr2=0x7f4380032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 --2- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f438002ff60 0x7f4380032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f437c0096a0 tx=0x7f437c006b40).stop 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 msgr2=0x7f43901066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 --2- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 0x7f43901066e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f4384013510 tx=0x7f438400b040).stop 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 shutdown_connections 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 --2- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f438002ff60 0x7f4380032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 --2- 172.21.15.190:0/3085457004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4390060280 0x7f43901066e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 >> 172.21.15.190:0/3085457004 conn(0x7f43900f0b70 msgr2=0x7f43900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:54.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 shutdown_connections 2024-01-16T17:14:54.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:54.184+0000 7f4396d6a700 1 -- 172.21.15.190:0/3085457004 wait complete. 2024-01-16T17:14:54.165 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:54.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:54 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:54.183758+0000 mon.smithi154 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.190:0/3085457004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:55.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:55.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:55.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:53.820743+0000 mgr.smithi154.sllivz (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.684+0000 7f63b8036700 1 -- 172.21.15.190:0/4102914762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 msgr2=0x7f63b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.684+0000 7f63b8036700 1 --2- 172.21.15.190:0/4102914762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 0x7f63b00f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f63a0009fb0 tx=0x7f63a0009b30).stop 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.684+0000 7f63b8036700 1 -- 172.21.15.190:0/4102914762 shutdown_connections 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.684+0000 7f63b8036700 1 --2- 172.21.15.190:0/4102914762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 0x7f63b00f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.684+0000 7f63b8036700 1 -- 172.21.15.190:0/4102914762 >> 172.21.15.190:0/4102914762 conn(0x7f63b00f0b70 msgr2=0x7f63b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.684+0000 7f63b8036700 1 -- 172.21.15.190:0/4102914762 shutdown_connections 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.684+0000 7f63b8036700 1 -- 172.21.15.190:0/4102914762 wait complete. 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.685+0000 7f63b8036700 1 Processor -- start 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.685+0000 7f63b8036700 1 -- start start 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.685+0000 7f63b8036700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 0x7f63b00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.685+0000 7f63b8036700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b00fb820 con 0x7f63b00f5410 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.685+0000 7f63b5dd2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 0x7f63b00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.685+0000 7f63b5dd2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 0x7f63b00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50240/0 (socket says 172.21.15.190:50240) 2024-01-16T17:14:56.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.685+0000 7f63b5dd2700 1 -- 172.21.15.190:0/2937992851 learned_addr learned my addr 172.21.15.190:0/2937992851 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:56.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.686+0000 7f63b5dd2700 1 -- 172.21.15.190:0/2937992851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a0005040 con 0x7f63b00f5410 2024-01-16T17:14:56.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.686+0000 7f63b5dd2700 1 --2- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 0x7f63b00fd3a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f63a0013510 tx=0x7f63a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:56.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.686+0000 7f63a67fc700 1 -- 172.21.15.190:0/2937992851 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63a0009320 con 0x7f63b00f5410 2024-01-16T17:14:56.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.686+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b00fb9c0 con 0x7f63b00f5410 2024-01-16T17:14:56.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.686+0000 7f63a67fc700 1 -- 172.21.15.190:0/2937992851 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63a0013f00 con 0x7f63b00f5410 2024-01-16T17:14:56.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.686+0000 7f63a67fc700 1 -- 172.21.15.190:0/2937992851 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63a0014d90 con 0x7f63b00f5410 2024-01-16T17:14:56.667 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.686+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b00fbe50 con 0x7f63b00f5410 2024-01-16T17:14:56.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.687+0000 7f63a67fc700 1 -- 172.21.15.190:0/2937992851 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63a00095f0 con 0x7f63b00f5410 2024-01-16T17:14:56.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.687+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b0059100 con 0x7f63b00f5410 2024-01-16T17:14:56.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.687+0000 7f63a67fc700 1 --2- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f639c02ff60 0x7f639c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:56.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.687+0000 7f63a67fc700 1 -- 172.21.15.190:0/2937992851 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63a0040fe0 con 0x7f63b00f5410 2024-01-16T17:14:56.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.687+0000 7f63b55d1700 1 --2- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f639c02ff60 0x7f639c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:56.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.687+0000 7f63b55d1700 1 --2- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f639c02ff60 0x7f639c032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f63ac0096a0 tx=0x7f63ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:56.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.692+0000 7f63a67fc700 1 -- 172.21.15.190:0/2937992851 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63a001d7f0 con 0x7f63b00f5410 2024-01-16T17:14:56.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.899+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63b00fae60 con 0x7f63b00f5410 2024-01-16T17:14:56.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.899+0000 7f63a67fc700 1 -- 172.21.15.190:0/2937992851 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f63a0010230 con 0x7f63b00f5410 2024-01-16T17:14:56.879 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:56.879 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f639c02ff60 msgr2=0x7f639c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 --2- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f639c02ff60 0x7f639c032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f63ac0096a0 tx=0x7f63ac006b40).stop 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 msgr2=0x7f63b00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 --2- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 0x7f63b00fd3a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f63a0013510 tx=0x7f63a000b040).stop 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 shutdown_connections 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 --2- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f639c02ff60 0x7f639c032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 --2- 172.21.15.190:0/2937992851 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63b00f5410 0x7f63b00fd3a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 >> 172.21.15.190:0/2937992851 conn(0x7f63b00f0b70 msgr2=0x7f63b00f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 shutdown_connections 2024-01-16T17:14:56.881 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:56.901+0000 7f63b8036700 1 -- 172.21.15.190:0/2937992851 wait complete. 2024-01-16T17:14:56.882 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:56.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:55.928583+0000 mon.smithi154 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:14:56.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:55.930171+0000 mon.smithi154 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:14:57.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:55.820995+0000 mgr.smithi154.sllivz (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:57.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:56.900268+0000 mon.smithi154 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.190:0/2937992851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:14:58.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:14:58.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:14:59.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 -- 172.21.15.190:0/3243227140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 msgr2=0x7f56580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:59.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 --2- 172.21.15.190:0/3243227140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 0x7f56580f55c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f5654009fb0 tx=0x7f5654009b30).stop 2024-01-16T17:14:59.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 -- 172.21.15.190:0/3243227140 shutdown_connections 2024-01-16T17:14:59.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 --2- 172.21.15.190:0/3243227140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 0x7f56580f55c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:59.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 -- 172.21.15.190:0/3243227140 >> 172.21.15.190:0/3243227140 conn(0x7f56580f09a0 msgr2=0x7f56580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 -- 172.21.15.190:0/3243227140 shutdown_connections 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 -- 172.21.15.190:0/3243227140 wait complete. 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 Processor -- start 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 -- start start 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 0x7f56580fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.363+0000 7f565e569700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56580fb5a0 con 0x7f56580f51a0 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f565d567700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 0x7f56580fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f565d567700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 0x7f56580fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50252/0 (socket says 172.21.15.190:50252) 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f565d567700 1 -- 172.21.15.190:0/1679630457 learned_addr learned my addr 172.21.15.190:0/1679630457 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f565d567700 1 -- 172.21.15.190:0/1679630457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5654005040 con 0x7f56580f51a0 2024-01-16T17:14:59.344 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f565d567700 1 --2- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 0x7f56580fd120 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5654013510 tx=0x7f565400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:59.345 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f564dffb700 1 -- 172.21.15.190:0/1679630457 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5654009320 con 0x7f56580f51a0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f564dffb700 1 -- 172.21.15.190:0/1679630457 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5654013f00 con 0x7f56580f51a0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56580fb740 con 0x7f56580f51a0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f564dffb700 1 -- 172.21.15.190:0/1679630457 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5654014dc0 con 0x7f56580f51a0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.364+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56580fbbd0 con 0x7f56580f51a0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.365+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5658059100 con 0x7f56580f51a0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.365+0000 7f564dffb700 1 -- 172.21.15.190:0/1679630457 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56540095f0 con 0x7f56580f51a0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.365+0000 7f564dffb700 1 --2- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f564402ff60 0x7f5644032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.365+0000 7f564dffb700 1 -- 172.21.15.190:0/1679630457 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56540410e0 con 0x7f56580f51a0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.366+0000 7f565cd66700 1 --2- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f564402ff60 0x7f5644032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:14:59.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.366+0000 7f565cd66700 1 --2- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f564402ff60 0x7f5644032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f56480096a0 tx=0x7f5648006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:14:59.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.370+0000 7f564dffb700 1 -- 172.21.15.190:0/1679630457 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5654014420 con 0x7f56580f51a0 2024-01-16T17:14:59.559 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.580+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5658045ad0 con 0x7f56580f51a0 2024-01-16T17:14:59.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.580+0000 7f564dffb700 1 -- 172.21.15.190:0/1679630457 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f565401c920 con 0x7f56580f51a0 2024-01-16T17:14:59.560 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:14:59.560 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f564402ff60 msgr2=0x7f5644032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 --2- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f564402ff60 0x7f5644032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f56480096a0 tx=0x7f5648006b40).stop 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 msgr2=0x7f56580fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 --2- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 0x7f56580fd120 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5654013510 tx=0x7f565400b040).stop 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 shutdown_connections 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 --2- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f564402ff60 0x7f5644032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 --2- 172.21.15.190:0/1679630457 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56580f51a0 0x7f56580fd120 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 >> 172.21.15.190:0/1679630457 conn(0x7f56580f09a0 msgr2=0x7f565805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 shutdown_connections 2024-01-16T17:14:59.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:14:59.582+0000 7f565e569700 1 -- 172.21.15.190:0/1679630457 wait complete. 2024-01-16T17:14:59.563 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:14:59.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:57.821344+0000 mgr.smithi154.sllivz (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:14:59.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:14:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:14:59.581434+0000 mon.smithi154 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.190:0/1679630457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:00.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:00.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:01.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:14:59.821675+0000 mgr.smithi154.sllivz (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:02.082 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 -- 172.21.15.190:0/1579300493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 msgr2=0x7f5ee80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 --2- 172.21.15.190:0/1579300493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 0x7f5ee80f55c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f5ee4009fb0 tx=0x7f5ee4009b30).stop 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 -- 172.21.15.190:0/1579300493 shutdown_connections 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 --2- 172.21.15.190:0/1579300493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 0x7f5ee80f55c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 -- 172.21.15.190:0/1579300493 >> 172.21.15.190:0/1579300493 conn(0x7f5ee80f09a0 msgr2=0x7f5ee80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 -- 172.21.15.190:0/1579300493 shutdown_connections 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 -- 172.21.15.190:0/1579300493 wait complete. 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 Processor -- start 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.100+0000 7f5eef422700 1 -- start start 2024-01-16T17:15:02.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5eef422700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 0x7f5ee81053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5eef422700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee81058d0 con 0x7f5ee80f51a0 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5eee420700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 0x7f5ee81053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5eee420700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 0x7f5ee81053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50262/0 (socket says 172.21.15.190:50262) 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5eee420700 1 -- 172.21.15.190:0/3364682892 learned_addr learned my addr 172.21.15.190:0/3364682892 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5eee420700 1 -- 172.21.15.190:0/3364682892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee4005040 con 0x7f5ee80f51a0 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5eee420700 1 --2- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 0x7f5ee81053e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f5ee4013510 tx=0x7f5ee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5edeffd700 1 -- 172.21.15.190:0/3364682892 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ee4009320 con 0x7f5ee80f51a0 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee8105ad0 con 0x7f5ee80f51a0 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5edeffd700 1 -- 172.21.15.190:0/3364682892 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ee4013f00 con 0x7f5ee80f51a0 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.101+0000 7f5edeffd700 1 -- 172.21.15.190:0/3364682892 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ee4014dc0 con 0x7f5ee80f51a0 2024-01-16T17:15:02.102 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.102+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee8105f60 con 0x7f5ee80f51a0 2024-01-16T17:15:02.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.102+0000 7f5edeffd700 1 -- 172.21.15.190:0/3364682892 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5ee40095f0 con 0x7f5ee80f51a0 2024-01-16T17:15:02.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.102+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee8045ad0 con 0x7f5ee80f51a0 2024-01-16T17:15:02.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.102+0000 7f5edeffd700 1 --2- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ed402ff60 0x7f5ed4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:02.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.102+0000 7f5edeffd700 1 -- 172.21.15.190:0/3364682892 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5ee4041100 con 0x7f5ee80f51a0 2024-01-16T17:15:02.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.103+0000 7f5eedc1f700 1 --2- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ed402ff60 0x7f5ed4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:02.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.103+0000 7f5eedc1f700 1 --2- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ed402ff60 0x7f5ed4032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5ed80096a0 tx=0x7f5ed8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:02.104 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.107+0000 7f5edeffd700 1 -- 172.21.15.190:0/3364682892 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ee4014420 con 0x7f5ee80f51a0 2024-01-16T17:15:02.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.312+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ee8106e10 con 0x7f5ee80f51a0 2024-01-16T17:15:02.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.312+0000 7f5edeffd700 1 -- 172.21.15.190:0/3364682892 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5ee4010230 con 0x7f5ee80f51a0 2024-01-16T17:15:02.293 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:02.293 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ed402ff60 msgr2=0x7f5ed4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 --2- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ed402ff60 0x7f5ed4032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f5ed80096a0 tx=0x7f5ed8006b40).stop 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 msgr2=0x7f5ee81053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 --2- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 0x7f5ee81053e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f5ee4013510 tx=0x7f5ee400b040).stop 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 shutdown_connections 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 --2- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ed402ff60 0x7f5ed4032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 --2- 172.21.15.190:0/3364682892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ee80f51a0 0x7f5ee81053e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 >> 172.21.15.190:0/3364682892 conn(0x7f5ee80f09a0 msgr2=0x7f5ee80f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:02.295 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 shutdown_connections 2024-01-16T17:15:02.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:02.314+0000 7f5eef422700 1 -- 172.21.15.190:0/3364682892 wait complete. 2024-01-16T17:15:02.296 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:02.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:02 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:02.314473+0000 mon.smithi154 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.190:0/3364682892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:03.675 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:03.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:03.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:01.821961+0000 mgr.smithi154.sllivz (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:04.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.884+0000 7f8e5332e700 1 -- 172.21.15.190:0/659613963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 msgr2=0x7f8e4c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:04.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.884+0000 7f8e5332e700 1 --2- 172.21.15.190:0/659613963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 0x7f8e4c0f9190 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f8e3c009fb0 tx=0x7f8e3c009b30).stop 2024-01-16T17:15:04.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.884+0000 7f8e5332e700 1 -- 172.21.15.190:0/659613963 shutdown_connections 2024-01-16T17:15:04.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.884+0000 7f8e5332e700 1 --2- 172.21.15.190:0/659613963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 0x7f8e4c0f9190 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:04.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.884+0000 7f8e5332e700 1 -- 172.21.15.190:0/659613963 >> 172.21.15.190:0/659613963 conn(0x7f8e4c0f0b70 msgr2=0x7f8e4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:04.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.884+0000 7f8e5332e700 1 -- 172.21.15.190:0/659613963 shutdown_connections 2024-01-16T17:15:04.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.884+0000 7f8e5332e700 1 -- 172.21.15.190:0/659613963 wait complete. 2024-01-16T17:15:04.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e5332e700 1 Processor -- start 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e5332e700 1 -- start start 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e5332e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 0x7f8e4c0f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e5332e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c0f5090 con 0x7f8e4c0601e0 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e510ca700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 0x7f8e4c0f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e510ca700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 0x7f8e4c0f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:32802/0 (socket says 172.21.15.190:32802) 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e510ca700 1 -- 172.21.15.190:0/2655384639 learned_addr learned my addr 172.21.15.190:0/2655384639 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e510ca700 1 -- 172.21.15.190:0/2655384639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3c005040 con 0x7f8e4c0601e0 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.885+0000 7f8e510ca700 1 --2- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 0x7f8e4c0f6c10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f8e3c004dd0 tx=0x7f8e3c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.886+0000 7f8e41ffb700 1 -- 172.21.15.190:0/2655384639 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e3c018070 con 0x7f8e4c0601e0 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.886+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c0f5230 con 0x7f8e4c0601e0 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.886+0000 7f8e41ffb700 1 -- 172.21.15.190:0/2655384639 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e3c013410 con 0x7f8e4c0601e0 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.886+0000 7f8e41ffb700 1 -- 172.21.15.190:0/2655384639 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e3c01c3c0 con 0x7f8e4c0601e0 2024-01-16T17:15:04.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.886+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c0f56c0 con 0x7f8e4c0601e0 2024-01-16T17:15:04.868 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.887+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c059100 con 0x7f8e4c0601e0 2024-01-16T17:15:04.868 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.887+0000 7f8e41ffb700 1 -- 172.21.15.190:0/2655384639 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e3c01d420 con 0x7f8e4c0601e0 2024-01-16T17:15:04.868 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.887+0000 7f8e41ffb700 1 --2- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8e3802ff60 0x7f8e38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:04.868 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.887+0000 7f8e41ffb700 1 -- 172.21.15.190:0/2655384639 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e3c041390 con 0x7f8e4c0601e0 2024-01-16T17:15:04.869 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.887+0000 7f8e508c9700 1 --2- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8e3802ff60 0x7f8e38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:04.869 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.887+0000 7f8e508c9700 1 --2- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8e3802ff60 0x7f8e38032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f8e480096a0 tx=0x7f8e48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:04.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:04.892+0000 7f8e41ffb700 1 -- 172.21.15.190:0/2655384639 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e3c01d820 con 0x7f8e4c0601e0 2024-01-16T17:15:05.078 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.098+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e4c045ad0 con 0x7f8e4c0601e0 2024-01-16T17:15:05.079 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.099+0000 7f8e41ffb700 1 -- 172.21.15.190:0/2655384639 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8e3c01b370 con 0x7f8e4c0601e0 2024-01-16T17:15:05.079 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:05.079 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8e3802ff60 msgr2=0x7f8e38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 --2- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8e3802ff60 0x7f8e38032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f8e480096a0 tx=0x7f8e48006b40).stop 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 msgr2=0x7f8e4c0f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 --2- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 0x7f8e4c0f6c10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f8e3c004dd0 tx=0x7f8e3c009b30).stop 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 shutdown_connections 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 --2- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8e3802ff60 0x7f8e38032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 --2- 172.21.15.190:0/2655384639 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8e4c0601e0 0x7f8e4c0f6c10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 >> 172.21.15.190:0/2655384639 conn(0x7f8e4c0f0b70 msgr2=0x7f8e4c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 shutdown_connections 2024-01-16T17:15:05.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:05.100+0000 7f8e5332e700 1 -- 172.21.15.190:0/2655384639 wait complete. 2024-01-16T17:15:05.082 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:05.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:03.822333+0000 mgr.smithi154.sllivz (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:05.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:05.100776+0000 mon.smithi154 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.190:0/2655384639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:06.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:06.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:07.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.523+0000 7fb24316f700 1 -- 172.21.15.190:0/385091519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f5410 msgr2=0x7fb23c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.523+0000 7fb24316f700 1 --2- 172.21.15.190:0/385091519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f5410 0x7fb23c0f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fb230009fb0 tx=0x7fb230009b30).stop 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.523+0000 7fb24316f700 1 -- 172.21.15.190:0/385091519 shutdown_connections 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.523+0000 7fb24316f700 1 --2- 172.21.15.190:0/385091519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f5410 0x7fb23c0f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.523+0000 7fb24316f700 1 -- 172.21.15.190:0/385091519 >> 172.21.15.190:0/385091519 conn(0x7fb23c0f0b70 msgr2=0x7fb23c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.523+0000 7fb24316f700 1 -- 172.21.15.190:0/385091519 shutdown_connections 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.523+0000 7fb24316f700 1 -- 172.21.15.190:0/385091519 wait complete. 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.524+0000 7fb24316f700 1 Processor -- start 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.524+0000 7fb24316f700 1 -- start start 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.524+0000 7fb24316f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c060200 0x7fb23c109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.524+0000 7fb24316f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c109a70 con 0x7fb23c060200 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.524+0000 7fb240f0b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c060200 0x7fb23c109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:07.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.524+0000 7fb240f0b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c060200 0x7fb23c109580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:32816/0 (socket says 172.21.15.190:32816) 2024-01-16T17:15:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.524+0000 7fb240f0b700 1 -- 172.21.15.190:0/3672401594 learned_addr learned my addr 172.21.15.190:0/3672401594 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.524+0000 7fb240f0b700 1 -- 172.21.15.190:0/3672401594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230005040 con 0x7fb23c060200 2024-01-16T17:15:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.525+0000 7fb240f0b700 1 --2- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c060200 0x7fb23c109580 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fb230009f80 tx=0x7fb230009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.525+0000 7fb2397fa700 1 -- 172.21.15.190:0/3672401594 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2300136c0 con 0x7fb23c060200 2024-01-16T17:15:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.525+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c109bb0 con 0x7fb23c060200 2024-01-16T17:15:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.525+0000 7fb2397fa700 1 -- 172.21.15.190:0/3672401594 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb230013e70 con 0x7fb23c060200 2024-01-16T17:15:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.525+0000 7fb2397fa700 1 -- 172.21.15.190:0/3672401594 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb230014420 con 0x7fb23c060200 2024-01-16T17:15:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.525+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c109fe0 con 0x7fb23c060200 2024-01-16T17:15:07.508 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.526+0000 7fb2397fa700 1 -- 172.21.15.190:0/3672401594 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb230025060 con 0x7fb23c060200 2024-01-16T17:15:07.508 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.526+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c045ad0 con 0x7fb23c060200 2024-01-16T17:15:07.508 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.526+0000 7fb2397fa700 1 --2- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22c02ff60 0x7fb22c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:07.508 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.526+0000 7fb2397fa700 1 -- 172.21.15.190:0/3672401594 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb230040e50 con 0x7fb23c060200 2024-01-16T17:15:07.508 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.526+0000 7fb23bfff700 1 --2- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22c02ff60 0x7fb22c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:07.508 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.526+0000 7fb23bfff700 1 --2- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22c02ff60 0x7fb22c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fb2280096a0 tx=0x7fb228006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:07.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.531+0000 7fb2397fa700 1 -- 172.21.15.190:0/3672401594 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb23001cd10 con 0x7fb23c060200 2024-01-16T17:15:07.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.736+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb23c0fae60 con 0x7fb23c060200 2024-01-16T17:15:07.717 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.736+0000 7fb2397fa700 1 -- 172.21.15.190:0/3672401594 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb23002d2d0 con 0x7fb23c060200 2024-01-16T17:15:07.717 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:07.717 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:07.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22c02ff60 msgr2=0x7fb22c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:07.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 --2- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22c02ff60 0x7fb22c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fb2280096a0 tx=0x7fb228006b40).stop 2024-01-16T17:15:07.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c060200 msgr2=0x7fb23c109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:07.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 --2- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c060200 0x7fb23c109580 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fb230009f80 tx=0x7fb230009b30).stop 2024-01-16T17:15:07.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 shutdown_connections 2024-01-16T17:15:07.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 --2- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22c02ff60 0x7fb22c032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:07.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 --2- 172.21.15.190:0/3672401594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c060200 0x7fb23c109580 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:07.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 >> 172.21.15.190:0/3672401594 conn(0x7fb23c0f0b70 msgr2=0x7fb23c0f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:07.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 shutdown_connections 2024-01-16T17:15:07.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:07.738+0000 7fb24316f700 1 -- 172.21.15.190:0/3672401594 wait complete. 2024-01-16T17:15:07.719 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:07.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:05.822660+0000 mgr.smithi154.sllivz (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:08.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:07.738398+0000 mon.smithi154 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.190:0/3672401594' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:09.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:09.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:09.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:07.822986+0000 mgr.smithi154.sllivz (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:10.182 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.202+0000 7f8008276700 1 -- 172.21.15.190:0/1650829401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 msgr2=0x7f80000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:10.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.202+0000 7f8008276700 1 --2- 172.21.15.190:0/1650829401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 0x7f80000f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f7ff4009fb0 tx=0x7f7ff4009b30).stop 2024-01-16T17:15:10.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.202+0000 7f8008276700 1 -- 172.21.15.190:0/1650829401 shutdown_connections 2024-01-16T17:15:10.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.202+0000 7f8008276700 1 --2- 172.21.15.190:0/1650829401 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 0x7f80000f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:10.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.202+0000 7f8008276700 1 -- 172.21.15.190:0/1650829401 >> 172.21.15.190:0/1650829401 conn(0x7f80000f1c70 msgr2=0x7f80000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:10.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.202+0000 7f8008276700 1 -- 172.21.15.190:0/1650829401 shutdown_connections 2024-01-16T17:15:10.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.202+0000 7f8008276700 1 -- 172.21.15.190:0/1650829401 wait complete. 2024-01-16T17:15:10.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.203+0000 7f8008276700 1 Processor -- start 2024-01-16T17:15:10.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.203+0000 7f8008276700 1 -- start start 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.203+0000 7f8008276700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 0x7f8000106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.203+0000 7f8008276700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8000106c20 con 0x7f80000f6510 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.203+0000 7f8006012700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 0x7f8000106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.203+0000 7f8006012700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 0x7f8000106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:32834/0 (socket says 172.21.15.190:32834) 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.203+0000 7f8006012700 1 -- 172.21.15.190:0/1832156175 learned_addr learned my addr 172.21.15.190:0/1832156175 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.203+0000 7f8006012700 1 -- 172.21.15.190:0/1832156175 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff4005040 con 0x7f80000f6510 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.204+0000 7f8006012700 1 --2- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 0x7f8000106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7ff4013510 tx=0x7f7ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.204+0000 7f7ff2ffd700 1 -- 172.21.15.190:0/1832156175 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ff4009320 con 0x7f80000f6510 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.204+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8000106e20 con 0x7f80000f6510 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.204+0000 7f7ff2ffd700 1 -- 172.21.15.190:0/1832156175 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ff4013f00 con 0x7f80000f6510 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.204+0000 7f7ff2ffd700 1 -- 172.21.15.190:0/1832156175 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ff4014d90 con 0x7f80000f6510 2024-01-16T17:15:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.204+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80001072b0 con 0x7f80000f6510 2024-01-16T17:15:10.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.204+0000 7f7ff2ffd700 1 -- 172.21.15.190:0/1832156175 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ff4014420 con 0x7f80000f6510 2024-01-16T17:15:10.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.205+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8000045ad0 con 0x7f80000f6510 2024-01-16T17:15:10.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.205+0000 7f7ff2ffd700 1 --2- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7fec02ff60 0x7f7fec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:10.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.205+0000 7f7ff2ffd700 1 -- 172.21.15.190:0/1832156175 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ff40410b0 con 0x7f80000f6510 2024-01-16T17:15:10.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.205+0000 7f8005811700 1 --2- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7fec02ff60 0x7f7fec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:10.186 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.205+0000 7f8005811700 1 --2- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7fec02ff60 0x7f7fec032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f7ffc0096a0 tx=0x7f7ffc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:10.190 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.210+0000 7f7ff2ffd700 1 -- 172.21.15.190:0/1832156175 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ff40095f0 con 0x7f80000f6510 2024-01-16T17:15:10.398 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.417+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80000fbf60 con 0x7f80000f6510 2024-01-16T17:15:10.399 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.418+0000 7f7ff2ffd700 1 -- 172.21.15.190:0/1832156175 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7ff401db70 con 0x7f80000f6510 2024-01-16T17:15:10.399 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:10.399 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:10.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7fec02ff60 msgr2=0x7f7fec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:10.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 --2- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7fec02ff60 0x7f7fec032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f7ffc0096a0 tx=0x7f7ffc006b40).stop 2024-01-16T17:15:10.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 msgr2=0x7f8000106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:10.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 --2- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 0x7f8000106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f7ff4013510 tx=0x7f7ff400b040).stop 2024-01-16T17:15:10.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 shutdown_connections 2024-01-16T17:15:10.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 --2- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7fec02ff60 0x7f7fec032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:10.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 --2- 172.21.15.190:0/1832156175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f80000f6510 0x7f8000106730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:10.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 >> 172.21.15.190:0/1832156175 conn(0x7f80000f1c70 msgr2=0x7f80000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:10.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 shutdown_connections 2024-01-16T17:15:10.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:10.420+0000 7f8008276700 1 -- 172.21.15.190:0/1832156175 wait complete. 2024-01-16T17:15:10.401 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:10.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:10 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:10.420089+0000 mon.smithi154 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.190:0/1832156175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:11.824 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:11.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:11.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:09.823338+0000 mgr.smithi154.sllivz (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:12.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 -- 172.21.15.190:0/1873839676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 msgr2=0x7f85780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:12.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 --2- 172.21.15.190:0/1873839676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 0x7f85780f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f8568009fb0 tx=0x7f8568009b30).stop 2024-01-16T17:15:12.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 -- 172.21.15.190:0/1873839676 shutdown_connections 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 --2- 172.21.15.190:0/1873839676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 0x7f85780f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 -- 172.21.15.190:0/1873839676 >> 172.21.15.190:0/1873839676 conn(0x7f85780f0b70 msgr2=0x7f85780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 -- 172.21.15.190:0/1873839676 shutdown_connections 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 -- 172.21.15.190:0/1873839676 wait complete. 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 Processor -- start 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 -- start start 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 0x7f8578108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.958+0000 7f8580762700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8578108e10 con 0x7f85780f5410 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f857e4fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 0x7f8578108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f857e4fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 0x7f8578108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36188/0 (socket says 172.21.15.190:36188) 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f857e4fe700 1 -- 172.21.15.190:0/1223557704 learned_addr learned my addr 172.21.15.190:0/1223557704 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f857e4fe700 1 -- 172.21.15.190:0/1223557704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8568005040 con 0x7f85780f5410 2024-01-16T17:15:12.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f857e4fe700 1 --2- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 0x7f8578108920 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8568014000 tx=0x7f856800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:12.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f856effd700 1 -- 172.21.15.190:0/1223557704 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85680095f0 con 0x7f85780f5410 2024-01-16T17:15:12.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8578109010 con 0x7f85780f5410 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f856effd700 1 -- 172.21.15.190:0/1223557704 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8568007e70 con 0x7f85780f5410 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f856effd700 1 -- 172.21.15.190:0/1223557704 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8568014ea0 con 0x7f85780f5410 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.959+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85781094a0 con 0x7f85780f5410 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.960+0000 7f856effd700 1 -- 172.21.15.190:0/1223557704 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8568025070 con 0x7f85780f5410 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.960+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8578045ad0 con 0x7f85780f5410 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.960+0000 7f856effd700 1 --2- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f856402ff60 0x7f8564032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.960+0000 7f856effd700 1 -- 172.21.15.190:0/1223557704 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8568040970 con 0x7f85780f5410 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.960+0000 7f857dcfd700 1 --2- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f856402ff60 0x7f8564032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:12.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.961+0000 7f857dcfd700 1 --2- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f856402ff60 0x7f8564032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f85740096a0 tx=0x7f8574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:12.944 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:12.965+0000 7f856effd700 1 -- 172.21.15.190:0/1223557704 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85680079c0 con 0x7f85780f5410 2024-01-16T17:15:13.148 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.169+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f857810a350 con 0x7f85780f5410 2024-01-16T17:15:13.149 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.170+0000 7f856effd700 1 -- 172.21.15.190:0/1223557704 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f856800f300 con 0x7f85780f5410 2024-01-16T17:15:13.150 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:13.150 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:13.151 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f856402ff60 msgr2=0x7f8564032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 --2- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f856402ff60 0x7f8564032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f85740096a0 tx=0x7f8574006b40).stop 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 msgr2=0x7f8578108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 --2- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 0x7f8578108920 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f8568014000 tx=0x7f856800b040).stop 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 shutdown_connections 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 --2- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f856402ff60 0x7f8564032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 --2- 172.21.15.190:0/1223557704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780f5410 0x7f8578108920 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 >> 172.21.15.190:0/1223557704 conn(0x7f85780f0b70 msgr2=0x7f857805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 shutdown_connections 2024-01-16T17:15:13.152 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:13.172+0000 7f8580762700 1 -- 172.21.15.190:0/1223557704 wait complete. 2024-01-16T17:15:13.153 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:13.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:11.823691+0000 mgr.smithi154.sllivz (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:13.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:13.170910+0000 mon.smithi154 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.190:0/1223557704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:14.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:14.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.648+0000 7fe8e8967700 1 -- 172.21.15.190:0/2456068383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 msgr2=0x7fe8e40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.648+0000 7fe8e8967700 1 --2- 172.21.15.190:0/2456068383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 0x7fe8e40f4be0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fe8d8009fb0 tx=0x7fe8d8009b30).stop 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.648+0000 7fe8e8967700 1 -- 172.21.15.190:0/2456068383 shutdown_connections 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.648+0000 7fe8e8967700 1 --2- 172.21.15.190:0/2456068383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 0x7fe8e40f4be0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.648+0000 7fe8e8967700 1 -- 172.21.15.190:0/2456068383 >> 172.21.15.190:0/2456068383 conn(0x7fe8e40f02e0 msgr2=0x7fe8e40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.648+0000 7fe8e8967700 1 -- 172.21.15.190:0/2456068383 shutdown_connections 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.648+0000 7fe8e8967700 1 -- 172.21.15.190:0/2456068383 wait complete. 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.649+0000 7fe8e8967700 1 Processor -- start 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.649+0000 7fe8e8967700 1 -- start start 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.649+0000 7fe8e8967700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 0x7fe8e4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.649+0000 7fe8e8967700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e4106bf0 con 0x7fe8e40f47c0 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.649+0000 7fe8e259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 0x7fe8e4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:15.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.649+0000 7fe8e259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 0x7fe8e4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36206/0 (socket says 172.21.15.190:36206) 2024-01-16T17:15:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.649+0000 7fe8e259c700 1 -- 172.21.15.190:0/2664488382 learned_addr learned my addr 172.21.15.190:0/2664488382 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.650+0000 7fe8e259c700 1 -- 172.21.15.190:0/2664488382 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d8005040 con 0x7fe8e40f47c0 2024-01-16T17:15:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.650+0000 7fe8e259c700 1 --2- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 0x7fe8e4106700 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe8d8013510 tx=0x7fe8d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.650+0000 7fe8d2ffd700 1 -- 172.21.15.190:0/2664488382 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d8009320 con 0x7fe8e40f47c0 2024-01-16T17:15:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.650+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e4106df0 con 0x7fe8e40f47c0 2024-01-16T17:15:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.650+0000 7fe8d2ffd700 1 -- 172.21.15.190:0/2664488382 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe8d8013f00 con 0x7fe8e40f47c0 2024-01-16T17:15:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.650+0000 7fe8d2ffd700 1 -- 172.21.15.190:0/2664488382 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8d8014d90 con 0x7fe8e40f47c0 2024-01-16T17:15:15.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.650+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e4107280 con 0x7fe8e40f47c0 2024-01-16T17:15:15.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.651+0000 7fe8d2ffd700 1 -- 172.21.15.190:0/2664488382 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8d80095f0 con 0x7fe8e40f47c0 2024-01-16T17:15:15.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.651+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e4045ad0 con 0x7fe8e40f47c0 2024-01-16T17:15:15.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.651+0000 7fe8d2ffd700 1 --2- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe8cc02ff60 0x7fe8cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:15.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.651+0000 7fe8d2ffd700 1 -- 172.21.15.190:0/2664488382 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8d80410c0 con 0x7fe8e40f47c0 2024-01-16T17:15:15.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.651+0000 7fe8e1d9b700 1 --2- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe8cc02ff60 0x7fe8cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:15.632 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.651+0000 7fe8e1d9b700 1 --2- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe8cc02ff60 0x7fe8cc032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe8d40096a0 tx=0x7fe8d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:15.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.656+0000 7fe8d2ffd700 1 -- 172.21.15.190:0/2664488382 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8d801d7f0 con 0x7fe8e40f47c0 2024-01-16T17:15:15.842 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.863+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8e4108130 con 0x7fe8e40f47c0 2024-01-16T17:15:15.843 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.864+0000 7fe8d2ffd700 1 -- 172.21.15.190:0/2664488382 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe8d8010230 con 0x7fe8e40f47c0 2024-01-16T17:15:15.843 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:15.843 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe8cc02ff60 msgr2=0x7fe8cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 --2- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe8cc02ff60 0x7fe8cc032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fe8d40096a0 tx=0x7fe8d4006b40).stop 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 msgr2=0x7fe8e4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 --2- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 0x7fe8e4106700 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fe8d8013510 tx=0x7fe8d800b040).stop 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 shutdown_connections 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 --2- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe8cc02ff60 0x7fe8cc032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 --2- 172.21.15.190:0/2664488382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8e40f47c0 0x7fe8e4106700 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 >> 172.21.15.190:0/2664488382 conn(0x7fe8e40f02e0 msgr2=0x7fe8e40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 shutdown_connections 2024-01-16T17:15:15.845 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:15.865+0000 7fe8e8967700 1 -- 172.21.15.190:0/2664488382 wait complete. 2024-01-16T17:15:15.846 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:15.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:13.824056+0000 mgr.smithi154.sllivz (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:16.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:16 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:15.864696+0000 mon.smithi154 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.190:0/2664488382' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:17.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:17.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:17.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:15.824367+0000 mgr.smithi154.sllivz (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:18.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.309+0000 7fb173cc1700 1 -- 172.21.15.190:0/2693490582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 msgr2=0x7fb16c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:18.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.309+0000 7fb173cc1700 1 --2- 172.21.15.190:0/2693490582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 0x7fb16c0f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fb15c009fb0 tx=0x7fb15c009b30).stop 2024-01-16T17:15:18.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.309+0000 7fb173cc1700 1 -- 172.21.15.190:0/2693490582 shutdown_connections 2024-01-16T17:15:18.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.309+0000 7fb173cc1700 1 --2- 172.21.15.190:0/2693490582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 0x7fb16c0f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:18.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.309+0000 7fb173cc1700 1 -- 172.21.15.190:0/2693490582 >> 172.21.15.190:0/2693490582 conn(0x7fb16c0f1c70 msgr2=0x7fb16c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:18.290 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.309+0000 7fb173cc1700 1 -- 172.21.15.190:0/2693490582 shutdown_connections 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.309+0000 7fb173cc1700 1 -- 172.21.15.190:0/2693490582 wait complete. 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.310+0000 7fb173cc1700 1 Processor -- start 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.310+0000 7fb173cc1700 1 -- start start 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.310+0000 7fb173cc1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 0x7fb16c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.310+0000 7fb173cc1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c069e20 con 0x7fb16c0f6510 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.310+0000 7fb171a5d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 0x7fb16c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.310+0000 7fb171a5d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 0x7fb16c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36216/0 (socket says 172.21.15.190:36216) 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.310+0000 7fb171a5d700 1 -- 172.21.15.190:0/86250409 learned_addr learned my addr 172.21.15.190:0/86250409 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.310+0000 7fb171a5d700 1 -- 172.21.15.190:0/86250409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c005040 con 0x7fb16c0f6510 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.311+0000 7fb171a5d700 1 --2- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 0x7fb16c06b9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fb15c014000 tx=0x7fb15c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.311+0000 7fb1627fc700 1 -- 172.21.15.190:0/86250409 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb15c0095f0 con 0x7fb16c0f6510 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.311+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb16c069fc0 con 0x7fb16c0f6510 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.311+0000 7fb1627fc700 1 -- 172.21.15.190:0/86250409 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb15c007e70 con 0x7fb16c0f6510 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.311+0000 7fb1627fc700 1 -- 172.21.15.190:0/86250409 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb15c014ea0 con 0x7fb16c0f6510 2024-01-16T17:15:18.291 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.311+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16c06a450 con 0x7fb16c0f6510 2024-01-16T17:15:18.292 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.311+0000 7fb1627fc700 1 -- 172.21.15.190:0/86250409 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb15c025070 con 0x7fb16c0f6510 2024-01-16T17:15:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.312+0000 7fb1627fc700 1 --2- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb15802ff60 0x7fb158032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.312+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16c059100 con 0x7fb16c0f6510 2024-01-16T17:15:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.312+0000 7fb1627fc700 1 -- 172.21.15.190:0/86250409 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb15c040a10 con 0x7fb16c0f6510 2024-01-16T17:15:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.312+0000 7fb17125c700 1 --2- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb15802ff60 0x7fb158032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:18.293 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.312+0000 7fb17125c700 1 --2- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb15802ff60 0x7fb158032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb1680096a0 tx=0x7fb168006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:18.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.316+0000 7fb1627fc700 1 -- 172.21.15.190:0/86250409 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb15c026350 con 0x7fb16c0f6510 2024-01-16T17:15:18.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.523+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb16c0fbf60 con 0x7fb16c0f6510 2024-01-16T17:15:18.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.524+0000 7fb1627fc700 1 -- 172.21.15.190:0/86250409 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb15c012330 con 0x7fb16c0f6510 2024-01-16T17:15:18.503 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:18.503 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb15802ff60 msgr2=0x7fb158032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 --2- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb15802ff60 0x7fb158032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fb1680096a0 tx=0x7fb168006b40).stop 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 msgr2=0x7fb16c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 --2- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 0x7fb16c06b9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fb15c014000 tx=0x7fb15c00b040).stop 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 shutdown_connections 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 --2- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb15802ff60 0x7fb158032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 --2- 172.21.15.190:0/86250409 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb16c0f6510 0x7fb16c06b9a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 >> 172.21.15.190:0/86250409 conn(0x7fb16c0f1c70 msgr2=0x7fb16c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 shutdown_connections 2024-01-16T17:15:18.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:18.525+0000 7fb173cc1700 1 -- 172.21.15.190:0/86250409 wait complete. 2024-01-16T17:15:18.506 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:19.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:17.824662+0000 mgr.smithi154.sllivz (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:19.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:18.524623+0000 mon.smithi154 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.190:0/86250409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:19.914 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:19.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 -- 172.21.15.190:0/3019404403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 msgr2=0x7f29e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 --2- 172.21.15.190:0/3019404403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 0x7f29e00f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f29d0009fb0 tx=0x7f29d0009b30).stop 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 -- 172.21.15.190:0/3019404403 shutdown_connections 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 --2- 172.21.15.190:0/3019404403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 0x7f29e00f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 -- 172.21.15.190:0/3019404403 >> 172.21.15.190:0/3019404403 conn(0x7f29e00f0b70 msgr2=0x7f29e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 -- 172.21.15.190:0/3019404403 shutdown_connections 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 -- 172.21.15.190:0/3019404403 wait complete. 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 Processor -- start 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 -- start start 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.062+0000 7f29e4fa6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 0x7f29e01055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:21.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29e4fa6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e0105ac0 con 0x7f29e00f5410 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29de59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 0x7f29e01055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29de59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 0x7f29e01055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36228/0 (socket says 172.21.15.190:36228) 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29de59c700 1 -- 172.21.15.190:0/1194023373 learned_addr learned my addr 172.21.15.190:0/1194023373 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29de59c700 1 -- 172.21.15.190:0/1194023373 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d0005040 con 0x7f29e00f5410 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29de59c700 1 --2- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 0x7f29e01055d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f29d0014000 tx=0x7f29d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29ceffd700 1 -- 172.21.15.190:0/1194023373 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d00095f0 con 0x7f29e00f5410 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e0105cc0 con 0x7f29e00f5410 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29ceffd700 1 -- 172.21.15.190:0/1194023373 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29d0007e70 con 0x7f29e00f5410 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29ceffd700 1 -- 172.21.15.190:0/1194023373 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d0014ea0 con 0x7f29e00f5410 2024-01-16T17:15:21.044 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.063+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e0106150 con 0x7f29e00f5410 2024-01-16T17:15:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.064+0000 7f29ceffd700 1 -- 172.21.15.190:0/1194023373 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29d0014690 con 0x7f29e00f5410 2024-01-16T17:15:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.064+0000 7f29ceffd700 1 --2- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f29c802ffb0 0x7f29c8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.064+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e0045ad0 con 0x7f29e00f5410 2024-01-16T17:15:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.064+0000 7f29ceffd700 1 -- 172.21.15.190:0/1194023373 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29d0040930 con 0x7f29e00f5410 2024-01-16T17:15:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.064+0000 7f29ddd9b700 1 --2- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f29c802ffb0 0x7f29c8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:21.045 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.065+0000 7f29ddd9b700 1 --2- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f29c802ffb0 0x7f29c8032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f29d40096a0 tx=0x7f29d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:21.048 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.069+0000 7f29ceffd700 1 -- 172.21.15.190:0/1194023373 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29d000f300 con 0x7f29e00f5410 2024-01-16T17:15:21.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.276+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29e0106fd0 con 0x7f29e00f5410 2024-01-16T17:15:21.256 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.277+0000 7f29ceffd700 1 -- 172.21.15.190:0/1194023373 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f29d000f300 con 0x7f29e00f5410 2024-01-16T17:15:21.257 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:21.257 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f29c802ffb0 msgr2=0x7f29c8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 --2- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f29c802ffb0 0x7f29c8032470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f29d40096a0 tx=0x7f29d4006b40).stop 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 msgr2=0x7f29e01055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 --2- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 0x7f29e01055d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f29d0014000 tx=0x7f29d000b040).stop 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 shutdown_connections 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 --2- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f29c802ffb0 0x7f29c8032470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 --2- 172.21.15.190:0/1194023373 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f29e00f5410 0x7f29e01055d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 >> 172.21.15.190:0/1194023373 conn(0x7f29e00f0b70 msgr2=0x7f29e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 shutdown_connections 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:21.279+0000 7f29e4fa6700 1 -- 172.21.15.190:0/1194023373 wait complete. 2024-01-16T17:15:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:21.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:19.824964+0000 mgr.smithi154.sllivz (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:21.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:21.278334+0000 mon.smithi154 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.190:0/1194023373' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:22.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:22.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:23.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:21.825303+0000 mgr.smithi154.sllivz (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:23.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.755+0000 7faad540d700 1 -- 172.21.15.190:0/1828797798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 msgr2=0x7faad00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:23.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.755+0000 7faad540d700 1 --2- 172.21.15.190:0/1828797798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 0x7faad00f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7faac4009fb0 tx=0x7faac4009b30).stop 2024-01-16T17:15:23.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.755+0000 7faad540d700 1 -- 172.21.15.190:0/1828797798 shutdown_connections 2024-01-16T17:15:23.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.755+0000 7faad540d700 1 --2- 172.21.15.190:0/1828797798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 0x7faad00f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:23.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.755+0000 7faad540d700 1 -- 172.21.15.190:0/1828797798 >> 172.21.15.190:0/1828797798 conn(0x7faad00f0b70 msgr2=0x7faad00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:23.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.755+0000 7faad540d700 1 -- 172.21.15.190:0/1828797798 shutdown_connections 2024-01-16T17:15:23.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.755+0000 7faad540d700 1 -- 172.21.15.190:0/1828797798 wait complete. 2024-01-16T17:15:23.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faad540d700 1 Processor -- start 2024-01-16T17:15:23.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faad540d700 1 -- start start 2024-01-16T17:15:23.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faad540d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 0x7faad0106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:23.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faad540d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad0106c40 con 0x7faad00f5410 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faaceffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 0x7faad0106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faaceffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 0x7faad0106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48324/0 (socket says 172.21.15.190:48324) 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faaceffd700 1 -- 172.21.15.190:0/3980844903 learned_addr learned my addr 172.21.15.190:0/3980844903 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faaceffd700 1 -- 172.21.15.190:0/3980844903 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac4005040 con 0x7faad00f5410 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.756+0000 7faaceffd700 1 --2- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 0x7faad0106750 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7faac4009f80 tx=0x7faac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.757+0000 7faab77fe700 1 -- 172.21.15.190:0/3980844903 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faac4009320 con 0x7faad00f5410 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.757+0000 7faab77fe700 1 -- 172.21.15.190:0/3980844903 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faac4013ec0 con 0x7faad00f5410 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.757+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad0106e40 con 0x7faad00f5410 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.757+0000 7faab77fe700 1 -- 172.21.15.190:0/3980844903 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faac401ece0 con 0x7faad00f5410 2024-01-16T17:15:23.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.757+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad01072d0 con 0x7faad00f5410 2024-01-16T17:15:23.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.758+0000 7faab77fe700 1 -- 172.21.15.190:0/3980844903 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faac401e420 con 0x7faad00f5410 2024-01-16T17:15:23.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.758+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad0045ad0 con 0x7faad00f5410 2024-01-16T17:15:23.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.758+0000 7faab77fe700 1 --2- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faab802ff60 0x7faab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:23.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.758+0000 7faab77fe700 1 -- 172.21.15.190:0/3980844903 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faac4040df0 con 0x7faad00f5410 2024-01-16T17:15:23.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.758+0000 7faace7fc700 1 --2- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faab802ff60 0x7faab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:23.738 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.759+0000 7faace7fc700 1 --2- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faab802ff60 0x7faab8032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7faac00096a0 tx=0x7faac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:23.742 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.763+0000 7faab77fe700 1 -- 172.21.15.190:0/3980844903 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faac401e6f0 con 0x7faad00f5410 2024-01-16T17:15:23.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.970+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faad0108180 con 0x7faad00f5410 2024-01-16T17:15:23.950 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.971+0000 7faab77fe700 1 -- 172.21.15.190:0/3980844903 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faac4004490 con 0x7faad00f5410 2024-01-16T17:15:23.950 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:23.950 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:23.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.972+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faab802ff60 msgr2=0x7faab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:23.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.972+0000 7faad540d700 1 --2- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faab802ff60 0x7faab8032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7faac00096a0 tx=0x7faac0006b40).stop 2024-01-16T17:15:23.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.972+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 msgr2=0x7faad0106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:23.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.972+0000 7faad540d700 1 --2- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 0x7faad0106750 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7faac4009f80 tx=0x7faac400b040).stop 2024-01-16T17:15:23.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.972+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 shutdown_connections 2024-01-16T17:15:23.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.972+0000 7faad540d700 1 --2- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7faab802ff60 0x7faab8032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:23.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.972+0000 7faad540d700 1 --2- 172.21.15.190:0/3980844903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faad00f5410 0x7faad0106750 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:23.952 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.972+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 >> 172.21.15.190:0/3980844903 conn(0x7faad00f0b70 msgr2=0x7faad005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:23.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.973+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 shutdown_connections 2024-01-16T17:15:23.953 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:23.973+0000 7faad540d700 1 -- 172.21.15.190:0/3980844903 wait complete. 2024-01-16T17:15:23.953 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:24.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:24 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:23.971846+0000 mon.smithi154 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.190:0/3980844903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:25.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:25.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:25.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:23.825662+0000 mgr.smithi154.sllivz (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:26.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.423+0000 7f160b991700 1 -- 172.21.15.190:0/1040140928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 msgr2=0x7f16040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:26.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.423+0000 7f160b991700 1 --2- 172.21.15.190:0/1040140928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 0x7f16040f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f15f4009fb0 tx=0x7f15f4009b30).stop 2024-01-16T17:15:26.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.423+0000 7f160b991700 1 -- 172.21.15.190:0/1040140928 shutdown_connections 2024-01-16T17:15:26.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.423+0000 7f160b991700 1 --2- 172.21.15.190:0/1040140928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 0x7f16040f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:26.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.423+0000 7f160b991700 1 -- 172.21.15.190:0/1040140928 >> 172.21.15.190:0/1040140928 conn(0x7f16040f0b70 msgr2=0x7f16040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.423+0000 7f160b991700 1 -- 172.21.15.190:0/1040140928 shutdown_connections 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.423+0000 7f160b991700 1 -- 172.21.15.190:0/1040140928 wait complete. 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.424+0000 7f160b991700 1 Processor -- start 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.424+0000 7f160b991700 1 -- start start 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.424+0000 7f160b991700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 0x7f1604105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.424+0000 7f160b991700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604105b20 con 0x7f16040f5410 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.424+0000 7f160972d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 0x7f1604105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.424+0000 7f160972d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 0x7f1604105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48340/0 (socket says 172.21.15.190:48340) 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.424+0000 7f160972d700 1 -- 172.21.15.190:0/2639757687 learned_addr learned my addr 172.21.15.190:0/2639757687 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.425+0000 7f160972d700 1 -- 172.21.15.190:0/2639757687 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f4005040 con 0x7f16040f5410 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.425+0000 7f160972d700 1 --2- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 0x7f1604105630 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f15f4013510 tx=0x7f15f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.425+0000 7f15f9ffb700 1 -- 172.21.15.190:0/2639757687 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15f4009320 con 0x7f16040f5410 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.425+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1604105d20 con 0x7f16040f5410 2024-01-16T17:15:26.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.425+0000 7f15f9ffb700 1 -- 172.21.15.190:0/2639757687 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15f4013f00 con 0x7f16040f5410 2024-01-16T17:15:26.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.425+0000 7f15f9ffb700 1 -- 172.21.15.190:0/2639757687 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15f4014d90 con 0x7f16040f5410 2024-01-16T17:15:26.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.425+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16041061b0 con 0x7f16040f5410 2024-01-16T17:15:26.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.426+0000 7f15f9ffb700 1 -- 172.21.15.190:0/2639757687 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15f40095f0 con 0x7f16040f5410 2024-01-16T17:15:26.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.426+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1604045ad0 con 0x7f16040f5410 2024-01-16T17:15:26.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.426+0000 7f15f9ffb700 1 --2- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f15f002ff60 0x7f15f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:26.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.426+0000 7f15f9ffb700 1 -- 172.21.15.190:0/2639757687 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15f4041080 con 0x7f16040f5410 2024-01-16T17:15:26.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.426+0000 7f1608f2c700 1 --2- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f15f002ff60 0x7f15f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:26.407 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.427+0000 7f1608f2c700 1 --2- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f15f002ff60 0x7f15f0032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f16000096a0 tx=0x7f1600006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:26.410 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.431+0000 7f15f9ffb700 1 -- 172.21.15.190:0/2639757687 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15f4014420 con 0x7f16040f5410 2024-01-16T17:15:26.614 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.635+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16040fae60 con 0x7f16040f5410 2024-01-16T17:15:26.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.636+0000 7f15f9ffb700 1 -- 172.21.15.190:0/2639757687 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f15f4010230 con 0x7f16040f5410 2024-01-16T17:15:26.615 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:26.615 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.637+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f15f002ff60 msgr2=0x7f15f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.637+0000 7f160b991700 1 --2- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f15f002ff60 0x7f15f0032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f16000096a0 tx=0x7f1600006b40).stop 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.637+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 msgr2=0x7f1604105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.637+0000 7f160b991700 1 --2- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 0x7f1604105630 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f15f4013510 tx=0x7f15f400b040).stop 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.637+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 shutdown_connections 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.637+0000 7f160b991700 1 --2- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f15f002ff60 0x7f15f0032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.637+0000 7f160b991700 1 --2- 172.21.15.190:0/2639757687 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16040f5410 0x7f1604105630 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.638+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 >> 172.21.15.190:0/2639757687 conn(0x7f16040f0b70 msgr2=0x7f16040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.638+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 shutdown_connections 2024-01-16T17:15:26.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:26.638+0000 7f160b991700 1 -- 172.21.15.190:0/2639757687 wait complete. 2024-01-16T17:15:26.618 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:27.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:25.826005+0000 mgr.smithi154.sllivz (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:27.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:26.636820+0000 mon.smithi154 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.190:0/2639757687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:27.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:27.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 -- 172.21.15.190:0/1183566289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 msgr2=0x7f73f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 --2- 172.21.15.190:0/1183566289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 0x7f73f80f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f73e8009fb0 tx=0x7f73e8009b30).stop 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 -- 172.21.15.190:0/1183566289 shutdown_connections 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 --2- 172.21.15.190:0/1183566289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 0x7f73f80f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 -- 172.21.15.190:0/1183566289 >> 172.21.15.190:0/1183566289 conn(0x7f73f80f0b70 msgr2=0x7f73f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 -- 172.21.15.190:0/1183566289 shutdown_connections 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 -- 172.21.15.190:0/1183566289 wait complete. 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 Processor -- start 2024-01-16T17:15:29.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 -- start start 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 0x7f73f80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.091+0000 7f74001c0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80fb820 con 0x7f73f80f5410 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f73fdf5c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 0x7f73f80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f73fdf5c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 0x7f73f80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48358/0 (socket says 172.21.15.190:48358) 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f73fdf5c700 1 -- 172.21.15.190:0/678939035 learned_addr learned my addr 172.21.15.190:0/678939035 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f73fdf5c700 1 -- 172.21.15.190:0/678939035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e8005040 con 0x7f73f80f5410 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f73fdf5c700 1 --2- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 0x7f73f80fd3a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f73e8013510 tx=0x7f73e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f73ee7fc700 1 -- 172.21.15.190:0/678939035 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73e8009320 con 0x7f73f80f5410 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f80fb9c0 con 0x7f73f80f5410 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f73ee7fc700 1 -- 172.21.15.190:0/678939035 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f73e8013f00 con 0x7f73f80f5410 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f73ee7fc700 1 -- 172.21.15.190:0/678939035 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f73e8014d90 con 0x7f73f80f5410 2024-01-16T17:15:29.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.092+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f80fbe50 con 0x7f73f80f5410 2024-01-16T17:15:29.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.093+0000 7f73ee7fc700 1 -- 172.21.15.190:0/678939035 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73e80095f0 con 0x7f73f80f5410 2024-01-16T17:15:29.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.093+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f8059100 con 0x7f73f80f5410 2024-01-16T17:15:29.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.093+0000 7f73ee7fc700 1 --2- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f73e402ff60 0x7f73e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:29.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.093+0000 7f73ee7fc700 1 -- 172.21.15.190:0/678939035 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73e8040fe0 con 0x7f73f80f5410 2024-01-16T17:15:29.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.093+0000 7f73fd75b700 1 --2- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f73e402ff60 0x7f73e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:29.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.094+0000 7f73fd75b700 1 --2- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f73e402ff60 0x7f73e4032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f73f40096a0 tx=0x7f73f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:29.077 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.098+0000 7f73ee7fc700 1 -- 172.21.15.190:0/678939035 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f73e801d7f0 con 0x7f73f80f5410 2024-01-16T17:15:29.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.304+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73f80fae60 con 0x7f73f80f5410 2024-01-16T17:15:29.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.305+0000 7f73ee7fc700 1 -- 172.21.15.190:0/678939035 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f73e8010230 con 0x7f73f80f5410 2024-01-16T17:15:29.284 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:29.284 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.306+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f73e402ff60 msgr2=0x7f73e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.306+0000 7f74001c0700 1 --2- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f73e402ff60 0x7f73e4032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f73f40096a0 tx=0x7f73f4006b40).stop 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.306+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 msgr2=0x7f73f80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.306+0000 7f74001c0700 1 --2- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 0x7f73f80fd3a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f73e8013510 tx=0x7f73e800b040).stop 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.306+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 shutdown_connections 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.307+0000 7f74001c0700 1 --2- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f73e402ff60 0x7f73e4032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.307+0000 7f74001c0700 1 --2- 172.21.15.190:0/678939035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73f80f5410 0x7f73f80fd3a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.307+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 >> 172.21.15.190:0/678939035 conn(0x7f73f80f0b70 msgr2=0x7f73f805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:29.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.307+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 shutdown_connections 2024-01-16T17:15:29.287 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:29.307+0000 7f74001c0700 1 -- 172.21.15.190:0/678939035 wait complete. 2024-01-16T17:15:29.287 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:29.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:27.826383+0000 mgr.smithi154.sllivz (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:29.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:29.305975+0000 mon.smithi154 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.190:0/678939035' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:30.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:30.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:31.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.712+0000 7fca9b59e700 1 -- 172.21.15.190:0/3355133562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 msgr2=0x7fca9c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:31.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.712+0000 7fca9b59e700 1 --2- 172.21.15.190:0/3355133562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 0x7fca9c0f66c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fca90009fb0 tx=0x7fca90009b30).stop 2024-01-16T17:15:31.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.712+0000 7fca9b59e700 1 -- 172.21.15.190:0/3355133562 shutdown_connections 2024-01-16T17:15:31.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.712+0000 7fca9b59e700 1 --2- 172.21.15.190:0/3355133562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 0x7fca9c0f66c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:31.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.712+0000 7fca9b59e700 1 -- 172.21.15.190:0/3355133562 >> 172.21.15.190:0/3355133562 conn(0x7fca9c0f1aa0 msgr2=0x7fca9c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.712+0000 7fca9b59e700 1 -- 172.21.15.190:0/3355133562 shutdown_connections 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.712+0000 7fca9b59e700 1 -- 172.21.15.190:0/3355133562 wait complete. 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.713+0000 7fca9b59e700 1 Processor -- start 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.713+0000 7fca9b59e700 1 -- start start 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.713+0000 7fca9b59e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 0x7fca9c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.713+0000 7fca9b59e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c069e20 con 0x7fca9c0f62a0 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.713+0000 7fca9a59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 0x7fca9c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.713+0000 7fca9a59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 0x7fca9c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48386/0 (socket says 172.21.15.190:48386) 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.713+0000 7fca9a59c700 1 -- 172.21.15.190:0/1264325752 learned_addr learned my addr 172.21.15.190:0/1264325752 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.714+0000 7fca9a59c700 1 -- 172.21.15.190:0/1264325752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca90005040 con 0x7fca9c0f62a0 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.714+0000 7fca9a59c700 1 --2- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 0x7fca9c06b9a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fca90013510 tx=0x7fca9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.714+0000 7fca8affd700 1 -- 172.21.15.190:0/1264325752 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca90009320 con 0x7fca9c0f62a0 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.714+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca9c069fc0 con 0x7fca9c0f62a0 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.714+0000 7fca8affd700 1 -- 172.21.15.190:0/1264325752 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fca90013f00 con 0x7fca9c0f62a0 2024-01-16T17:15:31.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.714+0000 7fca8affd700 1 -- 172.21.15.190:0/1264325752 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca90014dc0 con 0x7fca9c0f62a0 2024-01-16T17:15:31.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.714+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca9c06a450 con 0x7fca9c0f62a0 2024-01-16T17:15:31.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.715+0000 7fca8affd700 1 -- 172.21.15.190:0/1264325752 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca90014420 con 0x7fca9c0f62a0 2024-01-16T17:15:31.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.715+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9c059100 con 0x7fca9c0f62a0 2024-01-16T17:15:31.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.715+0000 7fca8affd700 1 --2- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fca8402ff60 0x7fca84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:31.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.715+0000 7fca8affd700 1 -- 172.21.15.190:0/1264325752 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca90041050 con 0x7fca9c0f62a0 2024-01-16T17:15:31.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.715+0000 7fca99d9b700 1 --2- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fca8402ff60 0x7fca84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:31.696 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.715+0000 7fca99d9b700 1 --2- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fca8402ff60 0x7fca84032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fca8c0096a0 tx=0x7fca8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:31.698 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.719+0000 7fca8affd700 1 -- 172.21.15.190:0/1264325752 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca900095f0 con 0x7fca9c0f62a0 2024-01-16T17:15:31.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:29.826720+0000 mgr.smithi154.sllivz (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:31.902 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.923+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca9c0fbce0 con 0x7fca9c0f62a0 2024-01-16T17:15:31.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.924+0000 7fca8affd700 1 -- 172.21.15.190:0/1264325752 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca9001cb10 con 0x7fca9c0f62a0 2024-01-16T17:15:31.903 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:31.903 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:31.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fca8402ff60 msgr2=0x7fca84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:31.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 --2- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fca8402ff60 0x7fca84032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fca8c0096a0 tx=0x7fca8c006b40).stop 2024-01-16T17:15:31.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 msgr2=0x7fca9c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:31.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 --2- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 0x7fca9c06b9a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fca90013510 tx=0x7fca9000b040).stop 2024-01-16T17:15:31.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 shutdown_connections 2024-01-16T17:15:31.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 --2- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fca8402ff60 0x7fca84032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:31.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 --2- 172.21.15.190:0/1264325752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fca9c0f62a0 0x7fca9c06b9a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:31.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 >> 172.21.15.190:0/1264325752 conn(0x7fca9c0f1aa0 msgr2=0x7fca9c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:31.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 shutdown_connections 2024-01-16T17:15:31.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:31.926+0000 7fca9b59e700 1 -- 172.21.15.190:0/1264325752 wait complete. 2024-01-16T17:15:31.906 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:32.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:32 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:31.924962+0000 mon.smithi154 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.190:0/1264325752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:33.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:33.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:33.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:31.827037+0000 mgr.smithi154.sllivz (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:34.420 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.441+0000 7f4d6f59e700 1 -- 172.21.15.190:0/487225820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 msgr2=0x7f4d700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:34.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.441+0000 7f4d6f59e700 1 --2- 172.21.15.190:0/487225820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 0x7f4d700f55c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f4d60009fb0 tx=0x7f4d60009b30).stop 2024-01-16T17:15:34.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.441+0000 7f4d6f59e700 1 -- 172.21.15.190:0/487225820 shutdown_connections 2024-01-16T17:15:34.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.441+0000 7f4d6f59e700 1 --2- 172.21.15.190:0/487225820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 0x7f4d700f55c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:34.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.441+0000 7f4d6f59e700 1 -- 172.21.15.190:0/487225820 >> 172.21.15.190:0/487225820 conn(0x7f4d700f09a0 msgr2=0x7f4d700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:34.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.441+0000 7f4d6f59e700 1 -- 172.21.15.190:0/487225820 shutdown_connections 2024-01-16T17:15:34.421 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.441+0000 7f4d6f59e700 1 -- 172.21.15.190:0/487225820 wait complete. 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6f59e700 1 Processor -- start 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6f59e700 1 -- start start 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6f59e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 0x7f4d701053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6f59e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701058a0 con 0x7f4d700f51a0 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 0x7f4d701053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6e59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 0x7f4d701053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36516/0 (socket says 172.21.15.190:36516) 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6e59c700 1 -- 172.21.15.190:0/2462704282 learned_addr learned my addr 172.21.15.190:0/2462704282 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6e59c700 1 -- 172.21.15.190:0/2462704282 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60005040 con 0x7f4d700f51a0 2024-01-16T17:15:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.442+0000 7f4d6e59c700 1 --2- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 0x7f4d701053b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f4d60009830 tx=0x7f4d6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.443+0000 7f4d5effd700 1 -- 172.21.15.190:0/2462704282 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d60009320 con 0x7f4d700f51a0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.443+0000 7f4d5effd700 1 -- 172.21.15.190:0/2462704282 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d600136c0 con 0x7f4d700f51a0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.443+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d70105aa0 con 0x7f4d700f51a0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.443+0000 7f4d5effd700 1 -- 172.21.15.190:0/2462704282 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6001ee90 con 0x7f4d700f51a0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.443+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d70105f30 con 0x7f4d700f51a0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.444+0000 7f4d5effd700 1 -- 172.21.15.190:0/2462704282 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d6001e420 con 0x7f4d700f51a0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.444+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70045ad0 con 0x7f4d700f51a0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.444+0000 7f4d5effd700 1 --2- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4d5802ff10 0x7f4d580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.444+0000 7f4d5effd700 1 -- 172.21.15.190:0/2462704282 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d60040f40 con 0x7f4d700f51a0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.444+0000 7f4d6dd9b700 1 --2- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4d5802ff10 0x7f4d580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:34.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.444+0000 7f4d6dd9b700 1 --2- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4d5802ff10 0x7f4d580323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f4d640096a0 tx=0x7f4d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:34.428 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.449+0000 7f4d5effd700 1 -- 172.21.15.190:0/2462704282 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d6006d2b0 con 0x7f4d700f51a0 2024-01-16T17:15:34.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.658+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d70106de0 con 0x7f4d700f51a0 2024-01-16T17:15:34.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.659+0000 7f4d5effd700 1 -- 172.21.15.190:0/2462704282 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4d600095f0 con 0x7f4d700f51a0 2024-01-16T17:15:34.638 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:34.639 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:34.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.660+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4d5802ff10 msgr2=0x7f4d580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:34.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.660+0000 7f4d6f59e700 1 --2- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4d5802ff10 0x7f4d580323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f4d640096a0 tx=0x7f4d64006b40).stop 2024-01-16T17:15:34.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.660+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 msgr2=0x7f4d701053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:34.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.660+0000 7f4d6f59e700 1 --2- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 0x7f4d701053b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f4d60009830 tx=0x7f4d6000b040).stop 2024-01-16T17:15:34.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.660+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 shutdown_connections 2024-01-16T17:15:34.640 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.660+0000 7f4d6f59e700 1 --2- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4d5802ff10 0x7f4d580323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:34.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.661+0000 7f4d6f59e700 1 --2- 172.21.15.190:0/2462704282 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d700f51a0 0x7f4d701053b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:34.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.661+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 >> 172.21.15.190:0/2462704282 conn(0x7f4d700f09a0 msgr2=0x7f4d700f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:34.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.661+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 shutdown_connections 2024-01-16T17:15:34.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:34.661+0000 7f4d6f59e700 1 -- 172.21.15.190:0/2462704282 wait complete. 2024-01-16T17:15:34.641 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:35.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:33.827333+0000 mgr.smithi154.sllivz (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:35.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:34.659930+0000 mon.smithi154 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.190:0/2462704282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:36.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:36.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:37.160 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/2519903709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 msgr2=0x7fb8a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 --2- 172.21.15.190:0/2519903709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 0x7fb8a80f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7fb898009fb0 tx=0x7fb898009b30).stop 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/2519903709 shutdown_connections 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 --2- 172.21.15.190:0/2519903709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 0x7fb8a80f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/2519903709 >> 172.21.15.190:0/2519903709 conn(0x7fb8a80f0b70 msgr2=0x7fb8a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/2519903709 shutdown_connections 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/2519903709 wait complete. 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 Processor -- start 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 -- start start 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 0x7fb8a8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.178+0000 7fb8ae7f2700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a8106c10 con 0x7fb8a80f5410 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8a7fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 0x7fb8a8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:37.161 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8a7fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 0x7fb8a8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36530/0 (socket says 172.21.15.190:36530) 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8a7fff700 1 -- 172.21.15.190:0/1444372343 learned_addr learned my addr 172.21.15.190:0/1444372343 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8a7fff700 1 -- 172.21.15.190:0/1444372343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb898005040 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8a7fff700 1 --2- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 0x7fb8a8106720 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fb898013510 tx=0x7fb89800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8a4ff9700 1 -- 172.21.15.190:0/1444372343 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb898009320 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8a4ff9700 1 -- 172.21.15.190:0/1444372343 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb898013f00 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a8106e10 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8a4ff9700 1 -- 172.21.15.190:0/1444372343 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb898014d90 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.179+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a81072a0 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.180+0000 7fb8a4ff9700 1 -- 172.21.15.190:0/1444372343 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb898014420 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.180+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a8045ad0 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.180+0000 7fb8a4ff9700 1 --2- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb89002ff60 0x7fb890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.180+0000 7fb8a4ff9700 1 -- 172.21.15.190:0/1444372343 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8980410c0 con 0x7fb8a80f5410 2024-01-16T17:15:37.162 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.180+0000 7fb8a77fe700 1 --2- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb89002ff60 0x7fb890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:37.163 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.181+0000 7fb8a77fe700 1 --2- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb89002ff60 0x7fb890032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb89c0096a0 tx=0x7fb89c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:37.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.185+0000 7fb8a4ff9700 1 -- 172.21.15.190:0/1444372343 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8980095f0 con 0x7fb8a80f5410 2024-01-16T17:15:37.370 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.389+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8a8108150 con 0x7fb8a80f5410 2024-01-16T17:15:37.370 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.390+0000 7fb8a4ff9700 1 -- 172.21.15.190:0/1444372343 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb89801db70 con 0x7fb8a80f5410 2024-01-16T17:15:37.371 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:37.371 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:37.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb89002ff60 msgr2=0x7fb890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:37.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 --2- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb89002ff60 0x7fb890032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fb89c0096a0 tx=0x7fb89c006b40).stop 2024-01-16T17:15:37.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 msgr2=0x7fb8a8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:37.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 --2- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 0x7fb8a8106720 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7fb898013510 tx=0x7fb89800b040).stop 2024-01-16T17:15:37.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 shutdown_connections 2024-01-16T17:15:37.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 --2- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb89002ff60 0x7fb890032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:37.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 --2- 172.21.15.190:0/1444372343 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8a80f5410 0x7fb8a8106720 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:37.372 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 >> 172.21.15.190:0/1444372343 conn(0x7fb8a80f0b70 msgr2=0x7fb8a805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:37.373 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 shutdown_connections 2024-01-16T17:15:37.373 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:37.391+0000 7fb8ae7f2700 1 -- 172.21.15.190:0/1444372343 wait complete. 2024-01-16T17:15:37.373 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:37.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:35.827643+0000 mgr.smithi154.sllivz (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:37.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:37.391754+0000 mon.smithi154 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.190:0/1444372343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:38.758 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:38.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:39.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:37.827961+0000 mgr.smithi154.sllivz (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:39.869 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.888+0000 7fa53e3d5700 1 -- 172.21.15.190:0/622725578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 msgr2=0x7fa5380fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.888+0000 7fa53e3d5700 1 --2- 172.21.15.190:0/622725578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 0x7fa5380fb550 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fa52c009fb0 tx=0x7fa52c009b30).stop 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.888+0000 7fa53e3d5700 1 -- 172.21.15.190:0/622725578 shutdown_connections 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.888+0000 7fa53e3d5700 1 --2- 172.21.15.190:0/622725578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 0x7fa5380fb550 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.888+0000 7fa53e3d5700 1 -- 172.21.15.190:0/622725578 >> 172.21.15.190:0/622725578 conn(0x7fa5380f0b70 msgr2=0x7fa5380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.888+0000 7fa53e3d5700 1 -- 172.21.15.190:0/622725578 shutdown_connections 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.888+0000 7fa53e3d5700 1 -- 172.21.15.190:0/622725578 wait complete. 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.888+0000 7fa53e3d5700 1 Processor -- start 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa53e3d5700 1 -- start start 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa53e3d5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 0x7fa5381088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa53e3d5700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa538108db0 con 0x7fa538060220 2024-01-16T17:15:39.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa537fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 0x7fa5381088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa537fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 0x7fa5381088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36552/0 (socket says 172.21.15.190:36552) 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa537fff700 1 -- 172.21.15.190:0/1438166498 learned_addr learned my addr 172.21.15.190:0/1438166498 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa537fff700 1 -- 172.21.15.190:0/1438166498 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c005040 con 0x7fa538060220 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa537fff700 1 --2- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 0x7fa5381088c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa52c013510 tx=0x7fa52c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa534ff9700 1 -- 172.21.15.190:0/1438166498 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa52c009320 con 0x7fa538060220 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa534ff9700 1 -- 172.21.15.190:0/1438166498 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa52c013f00 con 0x7fa538060220 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa538108fb0 con 0x7fa538060220 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa534ff9700 1 -- 172.21.15.190:0/1438166498 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa52c014dc0 con 0x7fa538060220 2024-01-16T17:15:39.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.889+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa538109440 con 0x7fa538060220 2024-01-16T17:15:39.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.890+0000 7fa534ff9700 1 -- 172.21.15.190:0/1438166498 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa52c014420 con 0x7fa538060220 2024-01-16T17:15:39.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.890+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa538045ad0 con 0x7fa538060220 2024-01-16T17:15:39.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.890+0000 7fa534ff9700 1 --2- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa52002ffb0 0x7fa520032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:39.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.890+0000 7fa534ff9700 1 -- 172.21.15.190:0/1438166498 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa52c0410b0 con 0x7fa538060220 2024-01-16T17:15:39.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.891+0000 7fa5377fe700 1 --2- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa52002ffb0 0x7fa520032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:39.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.891+0000 7fa5377fe700 1 --2- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa52002ffb0 0x7fa520032470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa5280096a0 tx=0x7fa528006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:39.875 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:39.895+0000 7fa534ff9700 1 -- 172.21.15.190:0/1438166498 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa52c06d2b0 con 0x7fa538060220 2024-01-16T17:15:40.084 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.104+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa53810a2f0 con 0x7fa538060220 2024-01-16T17:15:40.085 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.105+0000 7fa534ff9700 1 -- 172.21.15.190:0/1438166498 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa52c0095f0 con 0x7fa538060220 2024-01-16T17:15:40.085 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:40.085 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa52002ffb0 msgr2=0x7fa520032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 --2- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa52002ffb0 0x7fa520032470 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa5280096a0 tx=0x7fa528006b40).stop 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 msgr2=0x7fa5381088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 --2- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 0x7fa5381088c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa52c013510 tx=0x7fa52c00b040).stop 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 shutdown_connections 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 --2- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa52002ffb0 0x7fa520032470 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 --2- 172.21.15.190:0/1438166498 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa538060220 0x7fa5381088c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 >> 172.21.15.190:0/1438166498 conn(0x7fa5380f0b70 msgr2=0x7fa5380f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 shutdown_connections 2024-01-16T17:15:40.087 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:40.107+0000 7fa53e3d5700 1 -- 172.21.15.190:0/1438166498 wait complete. 2024-01-16T17:15:40.088 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:40.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:40 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:40.107147+0000 mon.smithi154 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.190:0/1438166498' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:41.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:41.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:41.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:39.828291+0000 mgr.smithi154.sllivz (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:42.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.568+0000 7fcb897a6700 1 -- 172.21.15.190:0/660099705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 msgr2=0x7fcb840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:42.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.568+0000 7fcb897a6700 1 --2- 172.21.15.190:0/660099705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 0x7fcb840f90f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fcb78009fb0 tx=0x7fcb78009b30).stop 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.568+0000 7fcb897a6700 1 -- 172.21.15.190:0/660099705 shutdown_connections 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.568+0000 7fcb897a6700 1 --2- 172.21.15.190:0/660099705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 0x7fcb840f90f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.568+0000 7fcb897a6700 1 -- 172.21.15.190:0/660099705 >> 172.21.15.190:0/660099705 conn(0x7fcb840f0b70 msgr2=0x7fcb840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.568+0000 7fcb897a6700 1 -- 172.21.15.190:0/660099705 shutdown_connections 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.568+0000 7fcb897a6700 1 -- 172.21.15.190:0/660099705 wait complete. 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.569+0000 7fcb897a6700 1 Processor -- start 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.569+0000 7fcb897a6700 1 -- start start 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.569+0000 7fcb897a6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 0x7fcb841066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.569+0000 7fcb897a6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb84106bd0 con 0x7fcb84060280 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.569+0000 7fcb82ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 0x7fcb841066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:42.551 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.569+0000 7fcb82ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 0x7fcb841066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52062/0 (socket says 172.21.15.190:52062) 2024-01-16T17:15:42.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.569+0000 7fcb82ffd700 1 -- 172.21.15.190:0/4221336198 learned_addr learned my addr 172.21.15.190:0/4221336198 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:42.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.570+0000 7fcb82ffd700 1 -- 172.21.15.190:0/4221336198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78005040 con 0x7fcb84060280 2024-01-16T17:15:42.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.570+0000 7fcb82ffd700 1 --2- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 0x7fcb841066e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fcb78013510 tx=0x7fcb7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:42.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.570+0000 7fcb6b7fe700 1 -- 172.21.15.190:0/4221336198 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb78009320 con 0x7fcb84060280 2024-01-16T17:15:42.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.570+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb84106dd0 con 0x7fcb84060280 2024-01-16T17:15:42.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.570+0000 7fcb6b7fe700 1 -- 172.21.15.190:0/4221336198 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb78013f00 con 0x7fcb84060280 2024-01-16T17:15:42.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.570+0000 7fcb6b7fe700 1 -- 172.21.15.190:0/4221336198 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb78014dc0 con 0x7fcb84060280 2024-01-16T17:15:42.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.570+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb84107260 con 0x7fcb84060280 2024-01-16T17:15:42.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.571+0000 7fcb6b7fe700 1 -- 172.21.15.190:0/4221336198 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb7801d3d0 con 0x7fcb84060280 2024-01-16T17:15:42.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.571+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb84045ad0 con 0x7fcb84060280 2024-01-16T17:15:42.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.571+0000 7fcb6b7fe700 1 --2- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb6c02ff60 0x7fcb6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:42.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.571+0000 7fcb6b7fe700 1 -- 172.21.15.190:0/4221336198 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb78041300 con 0x7fcb84060280 2024-01-16T17:15:42.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.571+0000 7fcb827fc700 1 --2- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb6c02ff60 0x7fcb6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:42.554 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.572+0000 7fcb827fc700 1 --2- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb6c02ff60 0x7fcb6c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcb740096a0 tx=0x7fcb74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:42.556 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.576+0000 7fcb6b7fe700 1 -- 172.21.15.190:0/4221336198 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb78012920 con 0x7fcb84060280 2024-01-16T17:15:42.763 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.783+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb841080b0 con 0x7fcb84060280 2024-01-16T17:15:42.764 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.783+0000 7fcb6b7fe700 1 -- 172.21.15.190:0/4221336198 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb780126f0 con 0x7fcb84060280 2024-01-16T17:15:42.764 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:42.764 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb6c02ff60 msgr2=0x7fcb6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 --2- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb6c02ff60 0x7fcb6c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fcb740096a0 tx=0x7fcb74006b40).stop 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 msgr2=0x7fcb841066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 --2- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 0x7fcb841066e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fcb78013510 tx=0x7fcb7800b040).stop 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 shutdown_connections 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 --2- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb6c02ff60 0x7fcb6c032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 --2- 172.21.15.190:0/4221336198 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb84060280 0x7fcb841066e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 >> 172.21.15.190:0/4221336198 conn(0x7fcb840f0b70 msgr2=0x7fcb840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:42.766 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 shutdown_connections 2024-01-16T17:15:42.767 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:42.785+0000 7fcb897a6700 1 -- 172.21.15.190:0/4221336198 wait complete. 2024-01-16T17:15:42.767 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:43.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:41.828610+0000 mgr.smithi154.sllivz (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:43.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:42.785473+0000 mon.smithi154 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.190:0/4221336198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:44.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:44.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:45.214 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.233+0000 7fe98567f700 1 -- 172.21.15.190:0/2852659630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 msgr2=0x7fe9800f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.233+0000 7fe98567f700 1 --2- 172.21.15.190:0/2852659630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 0x7fe9800f90f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fe974009fb0 tx=0x7fe974009b30).stop 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.233+0000 7fe98567f700 1 -- 172.21.15.190:0/2852659630 shutdown_connections 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.233+0000 7fe98567f700 1 --2- 172.21.15.190:0/2852659630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 0x7fe9800f90f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.233+0000 7fe98567f700 1 -- 172.21.15.190:0/2852659630 >> 172.21.15.190:0/2852659630 conn(0x7fe9800f0b70 msgr2=0x7fe9800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.233+0000 7fe98567f700 1 -- 172.21.15.190:0/2852659630 shutdown_connections 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.233+0000 7fe98567f700 1 -- 172.21.15.190:0/2852659630 wait complete. 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.234+0000 7fe98567f700 1 Processor -- start 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.234+0000 7fe98567f700 1 -- start start 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.234+0000 7fe98567f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 0x7fe980106690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:45.215 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.234+0000 7fe98567f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe980106b80 con 0x7fe980060280 2024-01-16T17:15:45.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.234+0000 7fe97effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 0x7fe980106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:45.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.234+0000 7fe97effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 0x7fe980106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52082/0 (socket says 172.21.15.190:52082) 2024-01-16T17:15:45.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.234+0000 7fe97effd700 1 -- 172.21.15.190:0/1180422410 learned_addr learned my addr 172.21.15.190:0/1180422410 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:45.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.235+0000 7fe97effd700 1 -- 172.21.15.190:0/1180422410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe974005040 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.235+0000 7fe97effd700 1 --2- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 0x7fe980106690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fe974013510 tx=0x7fe97400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.235+0000 7fe9677fe700 1 -- 172.21.15.190:0/1180422410 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe974009320 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.235+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe980106d80 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.235+0000 7fe9677fe700 1 -- 172.21.15.190:0/1180422410 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe974013f00 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.235+0000 7fe9677fe700 1 -- 172.21.15.190:0/1180422410 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe974014dc0 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.235+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe980107210 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.236+0000 7fe9677fe700 1 -- 172.21.15.190:0/1180422410 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9740095f0 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.236+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe980045ad0 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.236+0000 7fe9677fe700 1 --2- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe96802ffb0 0x7fe968032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.236+0000 7fe9677fe700 1 -- 172.21.15.190:0/1180422410 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9740410e0 con 0x7fe980060280 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.236+0000 7fe97e7fc700 1 --2- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe96802ffb0 0x7fe968032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:45.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.237+0000 7fe97e7fc700 1 --2- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe96802ffb0 0x7fe968032470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe9700096a0 tx=0x7fe970006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:45.221 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.241+0000 7fe9677fe700 1 -- 172.21.15.190:0/1180422410 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe974014420 con 0x7fe980060280 2024-01-16T17:15:45.429 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.449+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe980108060 con 0x7fe980060280 2024-01-16T17:15:45.429 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.449+0000 7fe9677fe700 1 -- 172.21.15.190:0/1180422410 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe97401c920 con 0x7fe980060280 2024-01-16T17:15:45.429 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:45.430 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:45.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe96802ffb0 msgr2=0x7fe968032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:45.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 --2- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe96802ffb0 0x7fe968032470 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe9700096a0 tx=0x7fe970006b40).stop 2024-01-16T17:15:45.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 msgr2=0x7fe980106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:45.431 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 --2- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 0x7fe980106690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fe974013510 tx=0x7fe97400b040).stop 2024-01-16T17:15:45.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 shutdown_connections 2024-01-16T17:15:45.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 --2- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe96802ffb0 0x7fe968032470 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:45.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 --2- 172.21.15.190:0/1180422410 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe980060280 0x7fe980106690 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:45.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 >> 172.21.15.190:0/1180422410 conn(0x7fe9800f0b70 msgr2=0x7fe9800f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:45.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 shutdown_connections 2024-01-16T17:15:45.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:45.451+0000 7fe98567f700 1 -- 172.21.15.190:0/1180422410 wait complete. 2024-01-16T17:15:45.432 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:45.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:43.828934+0000 mgr.smithi154.sllivz (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:46.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:46 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:45.451399+0000 mon.smithi154 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.190:0/1180422410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:46.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:46.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:47.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:45.829329+0000 mgr.smithi154.sllivz (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:47.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.926+0000 7f467a5f3700 1 -- 172.21.15.190:0/3631310051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 msgr2=0x7f46740f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:47.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.926+0000 7f467a5f3700 1 --2- 172.21.15.190:0/3631310051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 0x7f46740f90f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f4664009fb0 tx=0x7f4664009b30).stop 2024-01-16T17:15:47.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.926+0000 7f467a5f3700 1 -- 172.21.15.190:0/3631310051 shutdown_connections 2024-01-16T17:15:47.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.926+0000 7f467a5f3700 1 --2- 172.21.15.190:0/3631310051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 0x7f46740f90f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:47.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.926+0000 7f467a5f3700 1 -- 172.21.15.190:0/3631310051 >> 172.21.15.190:0/3631310051 conn(0x7f46740f0b70 msgr2=0x7f46740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:47.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.926+0000 7f467a5f3700 1 -- 172.21.15.190:0/3631310051 shutdown_connections 2024-01-16T17:15:47.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.926+0000 7f467a5f3700 1 -- 172.21.15.190:0/3631310051 wait complete. 2024-01-16T17:15:47.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.926+0000 7f467a5f3700 1 Processor -- start 2024-01-16T17:15:47.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f467a5f3700 1 -- start start 2024-01-16T17:15:47.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f467a5f3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 0x7f46740f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:47.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f467a5f3700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46740f4ff0 con 0x7f4674060280 2024-01-16T17:15:47.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f4673fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 0x7f46740f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:47.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f4673fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 0x7f46740f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52100/0 (socket says 172.21.15.190:52100) 2024-01-16T17:15:47.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f4673fff700 1 -- 172.21.15.190:0/3885016047 learned_addr learned my addr 172.21.15.190:0/3885016047 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:47.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f4673fff700 1 -- 172.21.15.190:0/3885016047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4664005040 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f4673fff700 1 --2- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 0x7f46740f6b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4664013510 tx=0x7f466400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f4670ff9700 1 -- 172.21.15.190:0/3885016047 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4664009320 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.927+0000 7f4670ff9700 1 -- 172.21.15.190:0/3885016047 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4664013f00 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.928+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46740f5190 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.928+0000 7f4670ff9700 1 -- 172.21.15.190:0/3885016047 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4664014dc0 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.928+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46740f5620 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.928+0000 7f4670ff9700 1 -- 172.21.15.190:0/3885016047 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4664014420 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.928+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4674059100 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.928+0000 7f4670ff9700 1 --2- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f466002ffb0 0x7f4660032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.928+0000 7f4670ff9700 1 -- 172.21.15.190:0/3885016047 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4664040ff0 con 0x7f4674060280 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.929+0000 7f46737fe700 1 --2- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f466002ffb0 0x7f4660032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:47.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.929+0000 7f46737fe700 1 --2- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f466002ffb0 0x7f4660032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f465c0096a0 tx=0x7f465c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:47.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:47.933+0000 7f4670ff9700 1 -- 172.21.15.190:0/3885016047 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f466406d2b0 con 0x7f4674060280 2024-01-16T17:15:48.117 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.138+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46740fbf50 con 0x7f4674060280 2024-01-16T17:15:48.118 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.139+0000 7f4670ff9700 1 -- 172.21.15.190:0/3885016047 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f46640095f0 con 0x7f4674060280 2024-01-16T17:15:48.118 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:48.118 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:48.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f466002ffb0 msgr2=0x7f4660032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 --2- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f466002ffb0 0x7f4660032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f465c0096a0 tx=0x7f465c006b40).stop 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 msgr2=0x7f46740f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 --2- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 0x7f46740f6b70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f4664013510 tx=0x7f466400b040).stop 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 shutdown_connections 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 --2- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f466002ffb0 0x7f4660032470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 --2- 172.21.15.190:0/3885016047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4674060280 0x7f46740f6b70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 >> 172.21.15.190:0/3885016047 conn(0x7f46740f0b70 msgr2=0x7f46740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 shutdown_connections 2024-01-16T17:15:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:48.140+0000 7f467a5f3700 1 -- 172.21.15.190:0/3885016047 wait complete. 2024-01-16T17:15:48.121 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:48.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:48 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:48.139690+0000 mon.smithi154 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.190:0/3885016047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:49.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:49.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:49.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:47.829590+0000 mgr.smithi154.sllivz (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:50.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 -- 172.21.15.190:0/1730365491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 msgr2=0x7fda900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:50.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 --2- 172.21.15.190:0/1730365491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 0x7fda900f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fda80009fb0 tx=0x7fda80009b30).stop 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 -- 172.21.15.190:0/1730365491 shutdown_connections 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 --2- 172.21.15.190:0/1730365491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 0x7fda900f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 -- 172.21.15.190:0/1730365491 >> 172.21.15.190:0/1730365491 conn(0x7fda900f0b70 msgr2=0x7fda900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 -- 172.21.15.190:0/1730365491 shutdown_connections 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 -- 172.21.15.190:0/1730365491 wait complete. 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 Processor -- start 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 -- start start 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 0x7fda900fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.560+0000 7fda97d55700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda900fb820 con 0x7fda900f5410 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda95af1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 0x7fda900fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda95af1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 0x7fda900fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52126/0 (socket says 172.21.15.190:52126) 2024-01-16T17:15:50.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda95af1700 1 -- 172.21.15.190:0/3932066671 learned_addr learned my addr 172.21.15.190:0/3932066671 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:50.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda95af1700 1 -- 172.21.15.190:0/3932066671 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda80005040 con 0x7fda900f5410 2024-01-16T17:15:50.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda95af1700 1 --2- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 0x7fda900fd3a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fda80014000 tx=0x7fda8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:50.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda867fc700 1 -- 172.21.15.190:0/3932066671 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda800095f0 con 0x7fda900f5410 2024-01-16T17:15:50.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda900fb9c0 con 0x7fda900f5410 2024-01-16T17:15:50.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda867fc700 1 -- 172.21.15.190:0/3932066671 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda80007e70 con 0x7fda900f5410 2024-01-16T17:15:50.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda867fc700 1 -- 172.21.15.190:0/3932066671 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda80014ea0 con 0x7fda900f5410 2024-01-16T17:15:50.542 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.561+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda900fbe50 con 0x7fda900f5410 2024-01-16T17:15:50.543 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.562+0000 7fda867fc700 1 -- 172.21.15.190:0/3932066671 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda80025070 con 0x7fda900f5410 2024-01-16T17:15:50.543 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.562+0000 7fda867fc700 1 --2- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda7c02ff60 0x7fda7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:50.543 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.562+0000 7fda867fc700 1 -- 172.21.15.190:0/3932066671 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda80040980 con 0x7fda900f5410 2024-01-16T17:15:50.543 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.562+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda90059100 con 0x7fda900f5410 2024-01-16T17:15:50.543 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.562+0000 7fda952f0700 1 --2- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda7c02ff60 0x7fda7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:50.543 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.563+0000 7fda952f0700 1 --2- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda7c02ff60 0x7fda7c032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fda8c0096a0 tx=0x7fda8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:50.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.567+0000 7fda867fc700 1 -- 172.21.15.190:0/3932066671 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda80026350 con 0x7fda900f5410 2024-01-16T17:15:50.750 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.771+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda90045ad0 con 0x7fda900f5410 2024-01-16T17:15:50.751 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.772+0000 7fda867fc700 1 -- 172.21.15.190:0/3932066671 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda90045ad0 con 0x7fda900f5410 2024-01-16T17:15:50.751 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:50.751 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda7c02ff60 msgr2=0x7fda7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 --2- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda7c02ff60 0x7fda7c032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fda8c0096a0 tx=0x7fda8c006b40).stop 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 msgr2=0x7fda900fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 --2- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 0x7fda900fd3a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fda80014000 tx=0x7fda8000b040).stop 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 shutdown_connections 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 --2- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda7c02ff60 0x7fda7c032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 --2- 172.21.15.190:0/3932066671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda900f5410 0x7fda900fd3a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 >> 172.21.15.190:0/3932066671 conn(0x7fda900f0b70 msgr2=0x7fda9005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:50.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 shutdown_connections 2024-01-16T17:15:50.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:50.773+0000 7fda97d55700 1 -- 172.21.15.190:0/3932066671 wait complete. 2024-01-16T17:15:50.754 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:51.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:49.829865+0000 mgr.smithi154.sllivz (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:51.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:50.772794+0000 mon.smithi154 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.190:0/3932066671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:52.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:52.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:53.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.280+0000 7f7144235700 1 -- 172.21.15.190:0/1357266372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 msgr2=0x7f713c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:53.260 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.280+0000 7f7144235700 1 --2- 172.21.15.190:0/1357266372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 0x7f713c0f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f7130009fb0 tx=0x7f7130009b30).stop 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.280+0000 7f7144235700 1 -- 172.21.15.190:0/1357266372 shutdown_connections 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.280+0000 7f7144235700 1 --2- 172.21.15.190:0/1357266372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 0x7f713c0f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.280+0000 7f7144235700 1 -- 172.21.15.190:0/1357266372 >> 172.21.15.190:0/1357266372 conn(0x7f713c0f0b70 msgr2=0x7f713c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.281+0000 7f7144235700 1 -- 172.21.15.190:0/1357266372 shutdown_connections 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.281+0000 7f7144235700 1 -- 172.21.15.190:0/1357266372 wait complete. 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.281+0000 7f7144235700 1 Processor -- start 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.281+0000 7f7144235700 1 -- start start 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.281+0000 7f7144235700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 0x7f713c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.281+0000 7f7144235700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c105b20 con 0x7f713c0f5410 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f7141fd1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 0x7f713c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f7141fd1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 0x7f713c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35930/0 (socket says 172.21.15.190:35930) 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f7141fd1700 1 -- 172.21.15.190:0/3883365937 learned_addr learned my addr 172.21.15.190:0/3883365937 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f7141fd1700 1 -- 172.21.15.190:0/3883365937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130005040 con 0x7f713c0f5410 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f7141fd1700 1 --2- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 0x7f713c105630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f7130013510 tx=0x7f713000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f712e7fc700 1 -- 172.21.15.190:0/3883365937 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7130009320 con 0x7f713c0f5410 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c105d20 con 0x7f713c0f5410 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f712e7fc700 1 -- 172.21.15.190:0/3883365937 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7130013f00 con 0x7f713c0f5410 2024-01-16T17:15:53.263 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f712e7fc700 1 -- 172.21.15.190:0/3883365937 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7130014d90 con 0x7f713c0f5410 2024-01-16T17:15:53.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.282+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c1061b0 con 0x7f713c0f5410 2024-01-16T17:15:53.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.283+0000 7f712e7fc700 1 -- 172.21.15.190:0/3883365937 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71300095f0 con 0x7f713c0f5410 2024-01-16T17:15:53.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.283+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713c045ad0 con 0x7f713c0f5410 2024-01-16T17:15:53.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.283+0000 7f712e7fc700 1 --2- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f712802ff60 0x7f7128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:53.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.283+0000 7f712e7fc700 1 -- 172.21.15.190:0/3883365937 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7130040fe0 con 0x7f713c0f5410 2024-01-16T17:15:53.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.284+0000 7f71417d0700 1 --2- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f712802ff60 0x7f7128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:53.264 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.284+0000 7f71417d0700 1 --2- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f712802ff60 0x7f7128032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f71380096a0 tx=0x7f7138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:53.267 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.288+0000 7f712e7fc700 1 -- 172.21.15.190:0/3883365937 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7130010230 con 0x7f713c0f5410 2024-01-16T17:15:53.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.495+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f713c0fae60 con 0x7f713c0f5410 2024-01-16T17:15:53.475 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.496+0000 7f712e7fc700 1 -- 172.21.15.190:0/3883365937 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7130010230 con 0x7f713c0f5410 2024-01-16T17:15:53.475 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:53.476 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:53.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f712802ff60 msgr2=0x7f7128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:53.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 --2- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f712802ff60 0x7f7128032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f71380096a0 tx=0x7f7138006b40).stop 2024-01-16T17:15:53.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 msgr2=0x7f713c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:53.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 --2- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 0x7f713c105630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f7130013510 tx=0x7f713000b040).stop 2024-01-16T17:15:53.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 shutdown_connections 2024-01-16T17:15:53.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 --2- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f712802ff60 0x7f7128032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:53.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 --2- 172.21.15.190:0/3883365937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f713c0f5410 0x7f713c105630 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:53.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 >> 172.21.15.190:0/3883365937 conn(0x7f713c0f0b70 msgr2=0x7f713c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:53.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 shutdown_connections 2024-01-16T17:15:53.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:53.498+0000 7f7144235700 1 -- 172.21.15.190:0/3883365937 wait complete. 2024-01-16T17:15:53.478 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:53.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:51.830263+0000 mgr.smithi154.sllivz (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:54.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:54 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:53.496925+0000 mon.smithi154 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.190:0/3883365937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:54.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:54.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:55.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:53.830616+0000 mgr.smithi154.sllivz (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:55.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.949+0000 7f86d7a38700 1 -- 172.21.15.190:0/2288179249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 msgr2=0x7f86d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:55.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.949+0000 7f86d7a38700 1 --2- 172.21.15.190:0/2288179249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 0x7f86d00f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f86c0009fb0 tx=0x7f86c0009b30).stop 2024-01-16T17:15:55.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.949+0000 7f86d7a38700 1 -- 172.21.15.190:0/2288179249 shutdown_connections 2024-01-16T17:15:55.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.949+0000 7f86d7a38700 1 --2- 172.21.15.190:0/2288179249 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 0x7f86d00f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:55.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.949+0000 7f86d7a38700 1 -- 172.21.15.190:0/2288179249 >> 172.21.15.190:0/2288179249 conn(0x7f86d00f1c70 msgr2=0x7f86d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:55.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.949+0000 7f86d7a38700 1 -- 172.21.15.190:0/2288179249 shutdown_connections 2024-01-16T17:15:55.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.949+0000 7f86d7a38700 1 -- 172.21.15.190:0/2288179249 wait complete. 2024-01-16T17:15:55.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.950+0000 7f86d7a38700 1 Processor -- start 2024-01-16T17:15:55.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.950+0000 7f86d7a38700 1 -- start start 2024-01-16T17:15:55.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.950+0000 7f86d7a38700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 0x7f86d006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.950+0000 7f86d7a38700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d0069e20 con 0x7f86d00f6510 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.950+0000 7f86d57d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 0x7f86d006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.950+0000 7f86d57d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 0x7f86d006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35948/0 (socket says 172.21.15.190:35948) 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.950+0000 7f86d57d4700 1 -- 172.21.15.190:0/2988002117 learned_addr learned my addr 172.21.15.190:0/2988002117 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.951+0000 7f86d57d4700 1 -- 172.21.15.190:0/2988002117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0005040 con 0x7f86d00f6510 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.951+0000 7f86d57d4700 1 --2- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 0x7f86d006b9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f86c0013510 tx=0x7f86c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.951+0000 7f86c5ffb700 1 -- 172.21.15.190:0/2988002117 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86c0009320 con 0x7f86d00f6510 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.951+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d0069fc0 con 0x7f86d00f6510 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.951+0000 7f86c5ffb700 1 -- 172.21.15.190:0/2988002117 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86c0013f00 con 0x7f86d00f6510 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.951+0000 7f86c5ffb700 1 -- 172.21.15.190:0/2988002117 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86c0014d90 con 0x7f86d00f6510 2024-01-16T17:15:55.931 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.951+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d006a450 con 0x7f86d00f6510 2024-01-16T17:15:55.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.952+0000 7f86c5ffb700 1 -- 172.21.15.190:0/2988002117 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86c0014420 con 0x7f86d00f6510 2024-01-16T17:15:55.932 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.952+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d0059100 con 0x7f86d00f6510 2024-01-16T17:15:55.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.952+0000 7f86c5ffb700 1 --2- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f86bc02ff60 0x7f86bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:55.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.952+0000 7f86c5ffb700 1 -- 172.21.15.190:0/2988002117 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86c0041000 con 0x7f86d00f6510 2024-01-16T17:15:55.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.952+0000 7f86d4fd3700 1 --2- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f86bc02ff60 0x7f86bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:55.933 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.953+0000 7f86d4fd3700 1 --2- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f86bc02ff60 0x7f86bc032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f86cc0096a0 tx=0x7f86cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:55.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:55.957+0000 7f86c5ffb700 1 -- 172.21.15.190:0/2988002117 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86c00095f0 con 0x7f86d00f6510 2024-01-16T17:15:56.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.164+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86d00fbf60 con 0x7f86d00f6510 2024-01-16T17:15:56.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.165+0000 7f86c5ffb700 1 -- 172.21.15.190:0/2988002117 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f86c001db70 con 0x7f86d00f6510 2024-01-16T17:15:56.144 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:56.144 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f86bc02ff60 msgr2=0x7f86bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 --2- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f86bc02ff60 0x7f86bc032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f86cc0096a0 tx=0x7f86cc006b40).stop 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 msgr2=0x7f86d006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 --2- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 0x7f86d006b9a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f86c0013510 tx=0x7f86c000b040).stop 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 shutdown_connections 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 --2- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f86bc02ff60 0x7f86bc032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 --2- 172.21.15.190:0/2988002117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f86d00f6510 0x7f86d006b9a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 >> 172.21.15.190:0/2988002117 conn(0x7f86d00f1c70 msgr2=0x7f86d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:56.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 shutdown_connections 2024-01-16T17:15:56.147 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:56.167+0000 7f86d7a38700 1 -- 172.21.15.190:0/2988002117 wait complete. 2024-01-16T17:15:56.147 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:56.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:55.929615+0000 mon.smithi154 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:15:56.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:55.930997+0000 mon.smithi154 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:15:56.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:56.166015+0000 mon.smithi154 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.190:0/2988002117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:15:57.524 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:15:57.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:15:57.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:55.830952+0000 mgr.smithi154.sllivz (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:58.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.615+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/1385250047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 msgr2=0x7fe5f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:58.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.615+0000 7fe5f5a5b700 1 --2- 172.21.15.190:0/1385250047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 0x7fe5f00f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fe5e4009fb0 tx=0x7fe5e4009b30).stop 2024-01-16T17:15:58.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.615+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/1385250047 shutdown_connections 2024-01-16T17:15:58.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.615+0000 7fe5f5a5b700 1 --2- 172.21.15.190:0/1385250047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 0x7fe5f00f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:58.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.615+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/1385250047 >> 172.21.15.190:0/1385250047 conn(0x7fe5f00f09a0 msgr2=0x7fe5f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:58.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.616+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/1385250047 shutdown_connections 2024-01-16T17:15:58.595 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.616+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/1385250047 wait complete. 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.616+0000 7fe5f5a5b700 1 Processor -- start 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.616+0000 7fe5f5a5b700 1 -- start start 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5f5a5b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5f5a5b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f0060240 con 0x7fe5f00f51a0 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5f4a59700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5f4a59700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35970/0 (socket says 172.21.15.190:35970) 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5f4a59700 1 -- 172.21.15.190:0/2009823311 learned_addr learned my addr 172.21.15.190:0/2009823311 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5f4a59700 1 -- 172.21.15.190:0/2009823311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e4005040 con 0x7fe5f00f51a0 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5f4a59700 1 --2- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fe5e4013510 tx=0x7fe5e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5ed7fa700 1 -- 172.21.15.190:0/2009823311 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5e4009320 con 0x7fe5f00f51a0 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5ed7fa700 1 -- 172.21.15.190:0/2009823311 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5e4013f00 con 0x7fe5f00f51a0 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.617+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5f0060440 con 0x7fe5f00f51a0 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.618+0000 7fe5ed7fa700 1 -- 172.21.15.190:0/2009823311 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5e4014dc0 con 0x7fe5f00f51a0 2024-01-16T17:15:58.597 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.618+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5f0109860 con 0x7fe5f00f51a0 2024-01-16T17:15:58.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.618+0000 7fe5ed7fa700 1 -- 172.21.15.190:0/2009823311 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5e401d3d0 con 0x7fe5f00f51a0 2024-01-16T17:15:58.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.618+0000 7fe5ed7fa700 1 --2- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe5e002ff60 0x7fe5e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:15:58.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.618+0000 7fe5ed7fa700 1 -- 172.21.15.190:0/2009823311 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe5e4041110 con 0x7fe5f00f51a0 2024-01-16T17:15:58.598 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.619+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5f0059100 con 0x7fe5f00f51a0 2024-01-16T17:15:58.599 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.619+0000 7fe5effff700 1 --2- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe5e002ff60 0x7fe5e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:15:58.599 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.619+0000 7fe5effff700 1 --2- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe5e002ff60 0x7fe5e0032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe5dc0096a0 tx=0x7fe5dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:15:58.602 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.623+0000 7fe5ed7fa700 1 -- 172.21.15.190:0/2009823311 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5e4012900 con 0x7fe5f00f51a0 2024-01-16T17:15:58.808 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.829+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5f00fabe0 con 0x7fe5f00f51a0 2024-01-16T17:15:58.809 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.830+0000 7fe5ed7fa700 1 -- 172.21.15.190:0/2009823311 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe5e402b020 con 0x7fe5f00f51a0 2024-01-16T17:15:58.809 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:15:58.809 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:15:58.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.831+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe5e002ff60 msgr2=0x7fe5e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.831+0000 7fe5f5a5b700 1 --2- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe5e002ff60 0x7fe5e0032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe5dc0096a0 tx=0x7fe5dc006b40).stop 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.831+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 msgr2=0x7fe5f01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.831+0000 7fe5f5a5b700 1 --2- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fe5e4013510 tx=0x7fe5e400b040).stop 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.832+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 shutdown_connections 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.832+0000 7fe5f5a5b700 1 --2- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe5e002ff60 0x7fe5e0032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.832+0000 7fe5f5a5b700 1 --2- 172.21.15.190:0/2009823311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5f00f51a0 0x7fe5f01091b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.832+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 >> 172.21.15.190:0/2009823311 conn(0x7fe5f00f09a0 msgr2=0x7fe5f00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.832+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 shutdown_connections 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:15:58.832+0000 7fe5f5a5b700 1 -- 172.21.15.190:0/2009823311 wait complete. 2024-01-16T17:15:58.811 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:15:59.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:57.831312+0000 mgr.smithi154.sllivz (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:15:59.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:15:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:15:58.830869+0000 mon.smithi154 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.190:0/2009823311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:00.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:00.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:01.230 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.249+0000 7f875e3cc700 1 -- 172.21.15.190:0/2222420103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 msgr2=0x7f87580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:01.230 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.249+0000 7f875e3cc700 1 --2- 172.21.15.190:0/2222420103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 0x7f87580f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f8748009fb0 tx=0x7f8748009b30).stop 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.249+0000 7f875e3cc700 1 -- 172.21.15.190:0/2222420103 shutdown_connections 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.249+0000 7f875e3cc700 1 --2- 172.21.15.190:0/2222420103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 0x7f87580f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.249+0000 7f875e3cc700 1 -- 172.21.15.190:0/2222420103 >> 172.21.15.190:0/2222420103 conn(0x7f87580f0b70 msgr2=0x7f87580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.249+0000 7f875e3cc700 1 -- 172.21.15.190:0/2222420103 shutdown_connections 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.249+0000 7f875e3cc700 1 -- 172.21.15.190:0/2222420103 wait complete. 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.250+0000 7f875e3cc700 1 Processor -- start 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.250+0000 7f875e3cc700 1 -- start start 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.250+0000 7f875e3cc700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 0x7f8758105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.250+0000 7f875e3cc700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758105b20 con 0x7f87580f5410 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f8757fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 0x7f8758105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f8757fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 0x7f8758105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35982/0 (socket says 172.21.15.190:35982) 2024-01-16T17:16:01.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f8757fff700 1 -- 172.21.15.190:0/1620647871 learned_addr learned my addr 172.21.15.190:0/1620647871 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f8757fff700 1 -- 172.21.15.190:0/1620647871 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8748005040 con 0x7f87580f5410 2024-01-16T17:16:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f8757fff700 1 --2- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 0x7f8758105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f8748013510 tx=0x7f874800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f8754ff9700 1 -- 172.21.15.190:0/1620647871 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8748009320 con 0x7f87580f5410 2024-01-16T17:16:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8758105d20 con 0x7f87580f5410 2024-01-16T17:16:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f8754ff9700 1 -- 172.21.15.190:0/1620647871 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8748013f00 con 0x7f87580f5410 2024-01-16T17:16:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f8754ff9700 1 -- 172.21.15.190:0/1620647871 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8748014d90 con 0x7f87580f5410 2024-01-16T17:16:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.251+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87581061b0 con 0x7f87580f5410 2024-01-16T17:16:01.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.252+0000 7f8754ff9700 1 -- 172.21.15.190:0/1620647871 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8748014420 con 0x7f87580f5410 2024-01-16T17:16:01.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.252+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8758045ad0 con 0x7f87580f5410 2024-01-16T17:16:01.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.252+0000 7f8754ff9700 1 --2- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f874002ff60 0x7f8740032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:01.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.252+0000 7f8754ff9700 1 -- 172.21.15.190:0/1620647871 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8748041070 con 0x7f87580f5410 2024-01-16T17:16:01.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.252+0000 7f87577fe700 1 --2- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f874002ff60 0x7f8740032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:01.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.253+0000 7f87577fe700 1 --2- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f874002ff60 0x7f8740032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f874c0096a0 tx=0x7f874c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:01.236 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.257+0000 7f8754ff9700 1 -- 172.21.15.190:0/1620647871 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f874806d2b0 con 0x7f87580f5410 2024-01-16T17:16:01.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.462+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87580fae60 con 0x7f87580f5410 2024-01-16T17:16:01.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.462+0000 7f8754ff9700 1 -- 172.21.15.190:0/1620647871 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f874801dd50 con 0x7f87580f5410 2024-01-16T17:16:01.442 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:01.442 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:01.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f874002ff60 msgr2=0x7f8740032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:01.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 --2- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f874002ff60 0x7f8740032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f874c0096a0 tx=0x7f874c006b40).stop 2024-01-16T17:16:01.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 msgr2=0x7f8758105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:01.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 --2- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 0x7f8758105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f8748013510 tx=0x7f874800b040).stop 2024-01-16T17:16:01.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 shutdown_connections 2024-01-16T17:16:01.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 --2- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f874002ff60 0x7f8740032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:01.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 --2- 172.21.15.190:0/1620647871 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87580f5410 0x7f8758105630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:01.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 >> 172.21.15.190:0/1620647871 conn(0x7f87580f0b70 msgr2=0x7f87580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:01.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 shutdown_connections 2024-01-16T17:16:01.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:01.464+0000 7f875e3cc700 1 -- 172.21.15.190:0/1620647871 wait complete. 2024-01-16T17:16:01.444 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:01.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:15:59.831666+0000 mgr.smithi154.sllivz (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:01.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:01.463420+0000 mon.smithi154 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.190:0/1620647871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:02.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:02.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:03.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:01.832015+0000 mgr.smithi154.sllivz (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:03.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.939+0000 7f8f50058700 1 -- 172.21.15.190:0/632708475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 msgr2=0x7f8f480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:03.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.939+0000 7f8f50058700 1 --2- 172.21.15.190:0/632708475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 0x7f8f480f6930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f8f38009fb0 tx=0x7f8f38009b30).stop 2024-01-16T17:16:03.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.939+0000 7f8f50058700 1 -- 172.21.15.190:0/632708475 shutdown_connections 2024-01-16T17:16:03.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.939+0000 7f8f50058700 1 --2- 172.21.15.190:0/632708475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 0x7f8f480f6930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:03.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.939+0000 7f8f50058700 1 -- 172.21.15.190:0/632708475 >> 172.21.15.190:0/632708475 conn(0x7f8f480f1c70 msgr2=0x7f8f480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:03.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.939+0000 7f8f50058700 1 -- 172.21.15.190:0/632708475 shutdown_connections 2024-01-16T17:16:03.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.939+0000 7f8f50058700 1 -- 172.21.15.190:0/632708475 wait complete. 2024-01-16T17:16:03.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.940+0000 7f8f50058700 1 Processor -- start 2024-01-16T17:16:03.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.940+0000 7f8f50058700 1 -- start start 2024-01-16T17:16:03.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.940+0000 7f8f50058700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 0x7f8f4806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:03.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.940+0000 7f8f50058700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f48069e20 con 0x7f8f480f6510 2024-01-16T17:16:03.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.940+0000 7f8f4ddf4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 0x7f8f4806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:03.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.940+0000 7f8f4ddf4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 0x7f8f4806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44362/0 (socket says 172.21.15.190:44362) 2024-01-16T17:16:03.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.940+0000 7f8f4ddf4700 1 -- 172.21.15.190:0/1957606649 learned_addr learned my addr 172.21.15.190:0/1957606649 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:03.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.941+0000 7f8f4ddf4700 1 -- 172.21.15.190:0/1957606649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f38005040 con 0x7f8f480f6510 2024-01-16T17:16:03.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.941+0000 7f8f4ddf4700 1 --2- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 0x7f8f4806b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f8f38013510 tx=0x7f8f3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:03.921 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.941+0000 7f8f3e7fc700 1 -- 172.21.15.190:0/1957606649 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f38009320 con 0x7f8f480f6510 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.941+0000 7f8f3e7fc700 1 -- 172.21.15.190:0/1957606649 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f38013f00 con 0x7f8f480f6510 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.941+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f48069fc0 con 0x7f8f480f6510 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.941+0000 7f8f3e7fc700 1 -- 172.21.15.190:0/1957606649 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f38014d90 con 0x7f8f480f6510 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.941+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4806a450 con 0x7f8f480f6510 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.942+0000 7f8f3e7fc700 1 -- 172.21.15.190:0/1957606649 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f380263a0 con 0x7f8f480f6510 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.942+0000 7f8f3e7fc700 1 --2- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3402ff60 0x7f8f34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.942+0000 7f8f3e7fc700 1 -- 172.21.15.190:0/1957606649 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f38040ff0 con 0x7f8f480f6510 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.942+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f48059100 con 0x7f8f480f6510 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.942+0000 7f8f4d5f3700 1 --2- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3402ff60 0x7f8f34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:03.922 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.943+0000 7f8f4d5f3700 1 --2- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3402ff60 0x7f8f34032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8f440096a0 tx=0x7f8f44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:03.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:03.947+0000 7f8f3e7fc700 1 -- 172.21.15.190:0/1957606649 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f38012920 con 0x7f8f480f6510 2024-01-16T17:16:04.132 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.153+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f480fbf60 con 0x7f8f480f6510 2024-01-16T17:16:04.133 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.154+0000 7f8f3e7fc700 1 -- 172.21.15.190:0/1957606649 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8f3802b020 con 0x7f8f480f6510 2024-01-16T17:16:04.133 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:04.133 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3402ff60 msgr2=0x7f8f34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 --2- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3402ff60 0x7f8f34032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8f440096a0 tx=0x7f8f44006b40).stop 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 msgr2=0x7f8f4806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 --2- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 0x7f8f4806b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f8f38013510 tx=0x7f8f3800b040).stop 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 shutdown_connections 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 --2- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8f3402ff60 0x7f8f34032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 --2- 172.21.15.190:0/1957606649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f480f6510 0x7f8f4806b9a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 >> 172.21.15.190:0/1957606649 conn(0x7f8f480f1c70 msgr2=0x7f8f480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 shutdown_connections 2024-01-16T17:16:04.135 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:04.156+0000 7f8f50058700 1 -- 172.21.15.190:0/1957606649 wait complete. 2024-01-16T17:16:04.136 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:04.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:04 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:04.155123+0000 mon.smithi154 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.190:0/1957606649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:05.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:05.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:05.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:03.832378+0000 mgr.smithi154.sllivz (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:06.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 -- 172.21.15.190:0/1812029278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 msgr2=0x7f78700fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:06.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 --2- 172.21.15.190:0/1812029278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 0x7f78700fb570 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f7858009fb0 tx=0x7f7858009b30).stop 2024-01-16T17:16:06.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 -- 172.21.15.190:0/1812029278 shutdown_connections 2024-01-16T17:16:06.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 --2- 172.21.15.190:0/1812029278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 0x7f78700fb570 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 -- 172.21.15.190:0/1812029278 >> 172.21.15.190:0/1812029278 conn(0x7f78700f0b70 msgr2=0x7f78700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 -- 172.21.15.190:0/1812029278 shutdown_connections 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 -- 172.21.15.190:0/1812029278 wait complete. 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 Processor -- start 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.681+0000 7f7875bdf700 1 -- start start 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f7875bdf700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 0x7f78701088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f7875bdf700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7870108dc0 con 0x7f7870060200 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f786f7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 0x7f78701088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f786f7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 0x7f78701088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44384/0 (socket says 172.21.15.190:44384) 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f786f7fe700 1 -- 172.21.15.190:0/477504172 learned_addr learned my addr 172.21.15.190:0/477504172 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f786f7fe700 1 -- 172.21.15.190:0/477504172 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7858005040 con 0x7f7870060200 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f786f7fe700 1 --2- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 0x7f78701088d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7858013510 tx=0x7f785800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:06.662 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f7867fff700 1 -- 172.21.15.190:0/477504172 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7858009320 con 0x7f7870060200 2024-01-16T17:16:06.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f7867fff700 1 -- 172.21.15.190:0/477504172 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7858013f00 con 0x7f7870060200 2024-01-16T17:16:06.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f7867fff700 1 -- 172.21.15.190:0/477504172 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7858014dc0 con 0x7f7870060200 2024-01-16T17:16:06.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7870108fc0 con 0x7f7870060200 2024-01-16T17:16:06.663 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.682+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7870109450 con 0x7f7870060200 2024-01-16T17:16:06.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.683+0000 7f7867fff700 1 -- 172.21.15.190:0/477504172 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f78580263a0 con 0x7f7870060200 2024-01-16T17:16:06.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.683+0000 7f7867fff700 1 --2- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f785c02ffb0 0x7f785c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:06.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.683+0000 7f7867fff700 1 -- 172.21.15.190:0/477504172 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7858041380 con 0x7f7870060200 2024-01-16T17:16:06.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.683+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7870045ad0 con 0x7f7870060200 2024-01-16T17:16:06.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.683+0000 7f786effd700 1 --2- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f785c02ffb0 0x7f785c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:06.664 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.684+0000 7f786effd700 1 --2- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f785c02ffb0 0x7f785c032470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f78600096a0 tx=0x7f7860006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:06.666 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.688+0000 7f7867fff700 1 -- 172.21.15.190:0/477504172 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f785801cb10 con 0x7f7870060200 2024-01-16T17:16:06.826 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:05.876022+0000 mon.smithi154 (mon.0) 228 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi154", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-16T17:16:06.827 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:05.878754+0000 mon.smithi154 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi154", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-16T17:16:06.827 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:05.879590+0000 mon.smithi154 (mon.0) 230 : audit [DBG] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-16T17:16:06.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.891+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f787010a170 con 0x7f7870060200 2024-01-16T17:16:06.870 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.892+0000 7f7867fff700 1 -- 172.21.15.190:0/477504172 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f785802b030 con 0x7f7870060200 2024-01-16T17:16:06.871 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:06.871 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:06.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f785c02ffb0 msgr2=0x7f785c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:06.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 --2- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f785c02ffb0 0x7f785c032470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f78600096a0 tx=0x7f7860006b40).stop 2024-01-16T17:16:06.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 msgr2=0x7f78701088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 --2- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 0x7f78701088d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7858013510 tx=0x7f785800b040).stop 2024-01-16T17:16:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 shutdown_connections 2024-01-16T17:16:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 --2- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f785c02ffb0 0x7f785c032470 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 --2- 172.21.15.190:0/477504172 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7870060200 0x7f78701088d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 >> 172.21.15.190:0/477504172 conn(0x7f78700f0b70 msgr2=0x7f78700f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 shutdown_connections 2024-01-16T17:16:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:06.893+0000 7f7875bdf700 1 -- 172.21.15.190:0/477504172 wait complete. 2024-01-16T17:16:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:07.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:05.832627+0000 mgr.smithi154.sllivz (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:07.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:16:05.874413+0000 mgr.smithi154.sllivz (mgr.14162) 157 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi154 ... 2024-01-16T17:16:07.708 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:11:44Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:12:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:12:59Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:14:13Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:14:51Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:15:28Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Traceback (most recent call last): 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-16T17:16:07.709 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: main() 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: r = ctx.func(ctx) 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: return func(ctx) 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: raise RuntimeError('uid/gid not found') 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: RuntimeError: uid/gid not found 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Traceback (most recent call last): 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: yield (conn, connr) 2024-01-16T17:16:07.710 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: code, '\n'.join(err))) 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi154 ... 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:11:44Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:12:21Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:12:59Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi154 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:14:13Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:14:51Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr time="2024-01-16T17:15:28Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-16T17:16:07.711 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Traceback (most recent call last): 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: main() 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: r = ctx.func(ctx) 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: return func(ctx) 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: raise RuntimeError('uid/gid not found') 2024-01-16T17:16:07.712 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: RuntimeError: uid/gid not found 2024-01-16T17:16:07.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:16:05.880185+0000 mgr.smithi154.sllivz (mgr.14162) 158 : cephadm [INF] Deploying daemon crash.smithi154 on smithi154 2024-01-16T17:16:07.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:06.892846+0000 mon.smithi154 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.190:0/477504172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:07.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:07.305839+0000 mon.smithi154 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:16:07.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:07.390611+0000 mon.smithi154 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:16:07.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:07.393036+0000 mon.smithi154 (mon.0) 234 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:16:07.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:07.393517+0000 mon.smithi154 (mon.0) 235 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-16T17:16:07.713 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:07.398003+0000 mon.smithi154 (mon.0) 236 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:16:08.301 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:08.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:08.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:07.393954+0000 mgr.smithi154.sllivz (mgr.14162) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-16T17:16:08.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:08 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:16:07.400990+0000 mgr.smithi154.sllivz (mgr.14162) 160 : cephadm [INF] Deploying daemon grafana.smithi154 on smithi154 2024-01-16T17:16:09.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.422+0000 7f84e2799700 1 -- 172.21.15.190:0/2312199048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 msgr2=0x7f84dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:09.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.422+0000 7f84e2799700 1 --2- 172.21.15.190:0/2312199048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 0x7f84dc0f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f84d0009fb0 tx=0x7f84d0009b30).stop 2024-01-16T17:16:09.401 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.422+0000 7f84e2799700 1 -- 172.21.15.190:0/2312199048 shutdown_connections 2024-01-16T17:16:09.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.422+0000 7f84e2799700 1 --2- 172.21.15.190:0/2312199048 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 0x7f84dc0f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:09.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.422+0000 7f84e2799700 1 -- 172.21.15.190:0/2312199048 >> 172.21.15.190:0/2312199048 conn(0x7f84dc0f0b70 msgr2=0x7f84dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:09.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.422+0000 7f84e2799700 1 -- 172.21.15.190:0/2312199048 shutdown_connections 2024-01-16T17:16:09.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.422+0000 7f84e2799700 1 -- 172.21.15.190:0/2312199048 wait complete. 2024-01-16T17:16:09.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.423+0000 7f84e2799700 1 Processor -- start 2024-01-16T17:16:09.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.423+0000 7f84e2799700 1 -- start start 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.423+0000 7f84e2799700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 0x7f84dc106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.423+0000 7f84e2799700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc106c00 con 0x7f84dc0f5410 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84dbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 0x7f84dc106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84dbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 0x7f84dc106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44410/0 (socket says 172.21.15.190:44410) 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84dbfff700 1 -- 172.21.15.190:0/2859225357 learned_addr learned my addr 172.21.15.190:0/2859225357 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84dbfff700 1 -- 172.21.15.190:0/2859225357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0005040 con 0x7f84dc0f5410 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84dbfff700 1 --2- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 0x7f84dc106710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f84d0013510 tx=0x7f84d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84d8ff9700 1 -- 172.21.15.190:0/2859225357 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84d0009320 con 0x7f84dc0f5410 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc106e00 con 0x7f84dc0f5410 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84d8ff9700 1 -- 172.21.15.190:0/2859225357 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84d0013f00 con 0x7f84dc0f5410 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc107290 con 0x7f84dc0f5410 2024-01-16T17:16:09.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.424+0000 7f84d8ff9700 1 -- 172.21.15.190:0/2859225357 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84d0014db0 con 0x7f84dc0f5410 2024-01-16T17:16:09.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.425+0000 7f84d8ff9700 1 -- 172.21.15.190:0/2859225357 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84d0014420 con 0x7f84dc0f5410 2024-01-16T17:16:09.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.425+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc045ad0 con 0x7f84dc0f5410 2024-01-16T17:16:09.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.425+0000 7f84d8ff9700 1 --2- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:09.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.425+0000 7f84d8ff9700 1 -- 172.21.15.190:0/2859225357 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84d0041070 con 0x7f84dc0f5410 2024-01-16T17:16:09.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.426+0000 7f84db7fe700 1 --2- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:09.406 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.426+0000 7f84db7fe700 1 --2- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f84cc0096a0 tx=0x7f84cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:09.411 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.430+0000 7f84d8ff9700 1 -- 172.21.15.190:0/2859225357 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84d00095f0 con 0x7f84dc0f5410 2024-01-16T17:16:09.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.635+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84dc0fae60 con 0x7f84dc0f5410 2024-01-16T17:16:09.615 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.636+0000 7f84d8ff9700 1 -- 172.21.15.190:0/2859225357 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f84d002b2d0 con 0x7f84dc0f5410 2024-01-16T17:16:09.615 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:09.615 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:09.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 msgr2=0x7f84c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:09.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 --2- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f84cc0096a0 tx=0x7f84cc006b40).stop 2024-01-16T17:16:09.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 msgr2=0x7f84dc106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:09.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 --2- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 0x7f84dc106710 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f84d0013510 tx=0x7f84d000b040).stop 2024-01-16T17:16:09.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 shutdown_connections 2024-01-16T17:16:09.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 --2- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:09.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 --2- 172.21.15.190:0/2859225357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f5410 0x7f84dc106710 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:09.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 >> 172.21.15.190:0/2859225357 conn(0x7f84dc0f0b70 msgr2=0x7f84dc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:09.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 shutdown_connections 2024-01-16T17:16:09.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:09.637+0000 7f84e2799700 1 -- 172.21.15.190:0/2859225357 wait complete. 2024-01-16T17:16:09.617 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:09.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:07.833013+0000 mgr.smithi154.sllivz (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:10.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:10 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:09.636494+0000 mon.smithi154 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.190:0/2859225357' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:11.007 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:11.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:11.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:09.833361+0000 mgr.smithi154.sllivz (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:12.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 -- 172.21.15.190:0/1373417923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 msgr2=0x7f1f140f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 --2- 172.21.15.190:0/1373417923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 0x7f1f140f4be0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f1f08009fb0 tx=0x7f1f08009b30).stop 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 -- 172.21.15.190:0/1373417923 shutdown_connections 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 --2- 172.21.15.190:0/1373417923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 0x7f1f140f4be0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 -- 172.21.15.190:0/1373417923 >> 172.21.15.190:0/1373417923 conn(0x7f1f140f02e0 msgr2=0x7f1f140f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 -- 172.21.15.190:0/1373417923 shutdown_connections 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 -- 172.21.15.190:0/1373417923 wait complete. 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 Processor -- start 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.155+0000 7f1f18fa6700 1 -- start start 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f18fa6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 0x7f1f14106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f18fa6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f14106bf0 con 0x7f1f140f47c0 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f1259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 0x7f1f14106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f1259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 0x7f1f14106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44428/0 (socket says 172.21.15.190:44428) 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f1259c700 1 -- 172.21.15.190:0/3559584690 learned_addr learned my addr 172.21.15.190:0/3559584690 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:12.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f1259c700 1 -- 172.21.15.190:0/3559584690 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f08005040 con 0x7f1f140f47c0 2024-01-16T17:16:12.138 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f1259c700 1 --2- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 0x7f1f14106700 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1f08013510 tx=0x7f1f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:12.138 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f02ffd700 1 -- 172.21.15.190:0/3559584690 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f08009320 con 0x7f1f140f47c0 2024-01-16T17:16:12.138 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f14106df0 con 0x7f1f140f47c0 2024-01-16T17:16:12.138 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f02ffd700 1 -- 172.21.15.190:0/3559584690 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f08013f00 con 0x7f1f140f47c0 2024-01-16T17:16:12.138 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f02ffd700 1 -- 172.21.15.190:0/3559584690 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f08014d90 con 0x7f1f140f47c0 2024-01-16T17:16:12.138 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.156+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f14107280 con 0x7f1f140f47c0 2024-01-16T17:16:12.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.157+0000 7f1f02ffd700 1 -- 172.21.15.190:0/3559584690 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f080263a0 con 0x7f1f140f47c0 2024-01-16T17:16:12.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.157+0000 7f1f02ffd700 1 --2- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1efc02ffb0 0x7f1efc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:12.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.157+0000 7f1f02ffd700 1 -- 172.21.15.190:0/3559584690 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f08041080 con 0x7f1f140f47c0 2024-01-16T17:16:12.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.157+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f14059100 con 0x7f1f140f47c0 2024-01-16T17:16:12.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.158+0000 7f1f11d9b700 1 --2- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1efc02ffb0 0x7f1efc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:12.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.158+0000 7f1f11d9b700 1 --2- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1efc02ffb0 0x7f1efc032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1f040096a0 tx=0x7f1f04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:12.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.162+0000 7f1f02ffd700 1 -- 172.21.15.190:0/3559584690 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f08012920 con 0x7f1f140f47c0 2024-01-16T17:16:12.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.366+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f14045ad0 con 0x7f1f140f47c0 2024-01-16T17:16:12.347 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.367+0000 7f1f02ffd700 1 -- 172.21.15.190:0/3559584690 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1f08012330 con 0x7f1f140f47c0 2024-01-16T17:16:12.347 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:12.347 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.368+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1efc02ffb0 msgr2=0x7f1efc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.368+0000 7f1f18fa6700 1 --2- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1efc02ffb0 0x7f1efc032470 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f1f040096a0 tx=0x7f1f04006b40).stop 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.368+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 msgr2=0x7f1f14106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.368+0000 7f1f18fa6700 1 --2- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 0x7f1f14106700 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f1f08013510 tx=0x7f1f0800b040).stop 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.368+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 shutdown_connections 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.368+0000 7f1f18fa6700 1 --2- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1efc02ffb0 0x7f1efc032470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.369+0000 7f1f18fa6700 1 --2- 172.21.15.190:0/3559584690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1f140f47c0 0x7f1f14106700 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.369+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 >> 172.21.15.190:0/3559584690 conn(0x7f1f140f02e0 msgr2=0x7f1f140f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.369+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 shutdown_connections 2024-01-16T17:16:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:12.369+0000 7f1f18fa6700 1 -- 172.21.15.190:0/3559584690 wait complete. 2024-01-16T17:16:12.350 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:12.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:12 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:12.368857+0000 mon.smithi154 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.190:0/3559584690' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:13.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:13.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:13.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:11.833646+0000 mgr.smithi154.sllivz (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/871408742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 msgr2=0x7f5ec00f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 --2- 172.21.15.190:0/871408742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 0x7f5ec00f7dc0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f5eb0009fb0 tx=0x7f5eb0009b30).stop 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/871408742 shutdown_connections 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 --2- 172.21.15.190:0/871408742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 0x7f5ec00f7dc0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/871408742 >> 172.21.15.190:0/871408742 conn(0x7f5ec00f0120 msgr2=0x7f5ec00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/871408742 shutdown_connections 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/871408742 wait complete. 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 Processor -- start 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.834+0000 7f5ec4f5a700 1 -- start start 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ec4f5a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 0x7f5ec00f7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ec4f5a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec00f56f0 con 0x7f5ec00f79a0 2024-01-16T17:16:14.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ebf7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 0x7f5ec00f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ebf7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 0x7f5ec00f7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46010/0 (socket says 172.21.15.190:46010) 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ebf7fe700 1 -- 172.21.15.190:0/3616371218 learned_addr learned my addr 172.21.15.190:0/3616371218 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ebf7fe700 1 -- 172.21.15.190:0/3616371218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0005040 con 0x7f5ec00f79a0 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ebf7fe700 1 --2- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 0x7f5ec00f7270 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f5eb0013510 tx=0x7f5eb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ea7fff700 1 -- 172.21.15.190:0/3616371218 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb0009320 con 0x7f5ec00f79a0 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ea7fff700 1 -- 172.21.15.190:0/3616371218 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5eb0013f00 con 0x7f5ec00f79a0 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec00f5890 con 0x7f5ec00f79a0 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ea7fff700 1 -- 172.21.15.190:0/3616371218 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5eb0014dc0 con 0x7f5ec00f79a0 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.835+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec00f5d20 con 0x7f5ec00f79a0 2024-01-16T17:16:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.836+0000 7f5ea7fff700 1 -- 172.21.15.190:0/3616371218 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5eb001d3d0 con 0x7f5ec00f79a0 2024-01-16T17:16:14.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.836+0000 7f5ea7fff700 1 --2- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ea802ff60 0x7f5ea8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:14.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.836+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec0059100 con 0x7f5ec00f79a0 2024-01-16T17:16:14.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.836+0000 7f5ea7fff700 1 -- 172.21.15.190:0/3616371218 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5eb00413b0 con 0x7f5ec00f79a0 2024-01-16T17:16:14.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.836+0000 7f5ebeffd700 1 --2- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ea802ff60 0x7f5ea8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:14.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.837+0000 7f5ebeffd700 1 --2- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ea802ff60 0x7f5ea8032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5eb40096a0 tx=0x7f5eb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:14.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:14.841+0000 7f5ea7fff700 1 -- 172.21.15.190:0/3616371218 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5eb001cb10 con 0x7f5ec00f79a0 2024-01-16T17:16:15.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.048+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ec00fbcf0 con 0x7f5ec00f79a0 2024-01-16T17:16:15.028 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.049+0000 7f5ea7fff700 1 -- 172.21.15.190:0/3616371218 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5eb001cb10 con 0x7f5ec00f79a0 2024-01-16T17:16:15.029 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:15.029 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:15.030 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.050+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ea802ff60 msgr2=0x7f5ea8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.050+0000 7f5ec4f5a700 1 --2- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ea802ff60 0x7f5ea8032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5eb40096a0 tx=0x7f5eb4006b40).stop 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.050+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 msgr2=0x7f5ec00f7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.050+0000 7f5ec4f5a700 1 --2- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 0x7f5ec00f7270 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f5eb0013510 tx=0x7f5eb000b040).stop 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.050+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 shutdown_connections 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.050+0000 7f5ec4f5a700 1 --2- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5ea802ff60 0x7f5ea8032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.050+0000 7f5ec4f5a700 1 --2- 172.21.15.190:0/3616371218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ec00f79a0 0x7f5ec00f7270 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.050+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 >> 172.21.15.190:0/3616371218 conn(0x7f5ec00f0120 msgr2=0x7f5ec00f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.051+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 shutdown_connections 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:15.051+0000 7f5ec4f5a700 1 -- 172.21.15.190:0/3616371218 wait complete. 2024-01-16T17:16:15.031 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:15.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:13.833961+0000 mgr.smithi154.sllivz (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:15.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:15.050849+0000 mon.smithi154 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.190:0/3616371218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:16.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:16.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:17.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.467+0000 7f42f0eff700 1 -- 172.21.15.190:0/341911308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 msgr2=0x7f42ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:17.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.467+0000 7f42f0eff700 1 --2- 172.21.15.190:0/341911308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 0x7f42ec0f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f42dc009fb0 tx=0x7f42dc009b30).stop 2024-01-16T17:16:17.449 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.467+0000 7f42f0eff700 1 -- 172.21.15.190:0/341911308 shutdown_connections 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.467+0000 7f42f0eff700 1 --2- 172.21.15.190:0/341911308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 0x7f42ec0f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.467+0000 7f42f0eff700 1 -- 172.21.15.190:0/341911308 >> 172.21.15.190:0/341911308 conn(0x7f42ec0f0b70 msgr2=0x7f42ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.467+0000 7f42f0eff700 1 -- 172.21.15.190:0/341911308 shutdown_connections 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.467+0000 7f42f0eff700 1 -- 172.21.15.190:0/341911308 wait complete. 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42f0eff700 1 Processor -- start 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42f0eff700 1 -- start start 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42f0eff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 0x7f42ec109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42f0eff700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec060240 con 0x7f42ec0f5410 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42ea59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 0x7f42ec109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42ea59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 0x7f42ec109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46034/0 (socket says 172.21.15.190:46034) 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42ea59c700 1 -- 172.21.15.190:0/3146043641 learned_addr learned my addr 172.21.15.190:0/3146043641 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42ea59c700 1 -- 172.21.15.190:0/3146043641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc005040 con 0x7f42ec0f5410 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42ea59c700 1 --2- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 0x7f42ec109430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f42dc013510 tx=0x7f42dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.468+0000 7f42e2ffd700 1 -- 172.21.15.190:0/3146043641 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42dc009320 con 0x7f42ec0f5410 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.469+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec060440 con 0x7f42ec0f5410 2024-01-16T17:16:17.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.469+0000 7f42e2ffd700 1 -- 172.21.15.190:0/3146043641 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f42dc013f00 con 0x7f42ec0f5410 2024-01-16T17:16:17.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.469+0000 7f42e2ffd700 1 -- 172.21.15.190:0/3146043641 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42dc014d90 con 0x7f42ec0f5410 2024-01-16T17:16:17.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.469+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec109ae0 con 0x7f42ec0f5410 2024-01-16T17:16:17.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.469+0000 7f42e2ffd700 1 -- 172.21.15.190:0/3146043641 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42dc0263a0 con 0x7f42ec0f5410 2024-01-16T17:16:17.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.469+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ec045ad0 con 0x7f42ec0f5410 2024-01-16T17:16:17.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.469+0000 7f42e2ffd700 1 --2- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f42d802ff60 0x7f42d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:17.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.469+0000 7f42e2ffd700 1 -- 172.21.15.190:0/3146043641 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42dc0413c0 con 0x7f42ec0f5410 2024-01-16T17:16:17.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.470+0000 7f42e9d9b700 1 --2- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f42d802ff60 0x7f42d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:17.451 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.470+0000 7f42e9d9b700 1 --2- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f42d802ff60 0x7f42d8032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f42d40096a0 tx=0x7f42d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:17.454 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.474+0000 7f42e2ffd700 1 -- 172.21.15.190:0/3146043641 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42dc012330 con 0x7f42ec0f5410 2024-01-16T17:16:17.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.678+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42ec0fae60 con 0x7f42ec0f5410 2024-01-16T17:16:17.658 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.678+0000 7f42e2ffd700 1 -- 172.21.15.190:0/3146043641 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f42dc01ccb0 con 0x7f42ec0f5410 2024-01-16T17:16:17.658 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:17.658 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:17.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f42d802ff60 msgr2=0x7f42d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:17.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 --2- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f42d802ff60 0x7f42d8032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f42d40096a0 tx=0x7f42d4006b40).stop 2024-01-16T17:16:17.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 msgr2=0x7f42ec109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:17.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 --2- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 0x7f42ec109430 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f42dc013510 tx=0x7f42dc00b040).stop 2024-01-16T17:16:17.660 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 shutdown_connections 2024-01-16T17:16:17.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 --2- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f42d802ff60 0x7f42d8032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:17.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 --2- 172.21.15.190:0/3146043641 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42ec0f5410 0x7f42ec109430 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:17.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 >> 172.21.15.190:0/3146043641 conn(0x7f42ec0f0b70 msgr2=0x7f42ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:17.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 shutdown_connections 2024-01-16T17:16:17.661 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:17.680+0000 7f42f0eff700 1 -- 172.21.15.190:0/3146043641 wait complete. 2024-01-16T17:16:17.661 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:17.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:15.834226+0000 mgr.smithi154.sllivz (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:18.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:18 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:17.680402+0000 mon.smithi154 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.190:0/3146043641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:19.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:19.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:19.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:17.834444+0000 mgr.smithi154.sllivz (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:20.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.162+0000 7f784830b700 1 -- 172.21.15.190:0/719655441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 msgr2=0x7f78400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:20.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.162+0000 7f784830b700 1 --2- 172.21.15.190:0/719655441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 0x7f78400f6930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f783c009fb0 tx=0x7f783c009b30).stop 2024-01-16T17:16:20.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.162+0000 7f784830b700 1 -- 172.21.15.190:0/719655441 shutdown_connections 2024-01-16T17:16:20.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.162+0000 7f784830b700 1 --2- 172.21.15.190:0/719655441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 0x7f78400f6930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:20.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.162+0000 7f784830b700 1 -- 172.21.15.190:0/719655441 >> 172.21.15.190:0/719655441 conn(0x7f78400f1c70 msgr2=0x7f78400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:20.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.162+0000 7f784830b700 1 -- 172.21.15.190:0/719655441 shutdown_connections 2024-01-16T17:16:20.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.162+0000 7f784830b700 1 -- 172.21.15.190:0/719655441 wait complete. 2024-01-16T17:16:20.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.162+0000 7f784830b700 1 Processor -- start 2024-01-16T17:16:20.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.163+0000 7f784830b700 1 -- start start 2024-01-16T17:16:20.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.163+0000 7f784830b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 0x7f7840106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:20.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.163+0000 7f784830b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7840106c20 con 0x7f78400f6510 2024-01-16T17:16:20.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.163+0000 7f78460a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 0x7f7840106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:20.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.163+0000 7f78460a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 0x7f7840106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:46056/0 (socket says 172.21.15.190:46056) 2024-01-16T17:16:20.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.163+0000 7f78460a7700 1 -- 172.21.15.190:0/3887290086 learned_addr learned my addr 172.21.15.190:0/3887290086 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:20.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.163+0000 7f78460a7700 1 -- 172.21.15.190:0/3887290086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c005040 con 0x7f78400f6510 2024-01-16T17:16:20.144 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.163+0000 7f78460a7700 1 --2- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 0x7f7840106730 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f783c013510 tx=0x7f783c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.164+0000 7f7836ffd700 1 -- 172.21.15.190:0/3887290086 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f783c009320 con 0x7f78400f6510 2024-01-16T17:16:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.164+0000 7f7836ffd700 1 -- 172.21.15.190:0/3887290086 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f783c013f00 con 0x7f78400f6510 2024-01-16T17:16:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.164+0000 7f7836ffd700 1 -- 172.21.15.190:0/3887290086 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f783c014d90 con 0x7f78400f6510 2024-01-16T17:16:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.164+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7840106e20 con 0x7f78400f6510 2024-01-16T17:16:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.164+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78401072b0 con 0x7f78400f6510 2024-01-16T17:16:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.165+0000 7f7836ffd700 1 -- 172.21.15.190:0/3887290086 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f783c0095f0 con 0x7f78400f6510 2024-01-16T17:16:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.165+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7840045ad0 con 0x7f78400f6510 2024-01-16T17:16:20.145 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.165+0000 7f7836ffd700 1 --2- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f782c02ff60 0x7f782c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:20.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.165+0000 7f7836ffd700 1 -- 172.21.15.190:0/3887290086 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f783c0410c0 con 0x7f78400f6510 2024-01-16T17:16:20.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.165+0000 7f78458a6700 1 --2- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f782c02ff60 0x7f782c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:20.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.165+0000 7f78458a6700 1 --2- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f782c02ff60 0x7f782c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f78300096a0 tx=0x7f7830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:20.150 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.171+0000 7f7836ffd700 1 -- 172.21.15.190:0/3887290086 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f783c01d7f0 con 0x7f78400f6510 2024-01-16T17:16:20.357 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.377+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f78400fbf60 con 0x7f78400f6510 2024-01-16T17:16:20.357 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.378+0000 7f7836ffd700 1 -- 172.21.15.190:0/3887290086 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f783c010230 con 0x7f78400f6510 2024-01-16T17:16:20.358 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:20.358 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:20.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f782c02ff60 msgr2=0x7f782c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:20.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 --2- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f782c02ff60 0x7f782c032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f78300096a0 tx=0x7f7830006b40).stop 2024-01-16T17:16:20.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 msgr2=0x7f7840106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:20.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 --2- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 0x7f7840106730 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f783c013510 tx=0x7f783c00b040).stop 2024-01-16T17:16:20.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 shutdown_connections 2024-01-16T17:16:20.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 --2- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f782c02ff60 0x7f782c032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:20.360 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 --2- 172.21.15.190:0/3887290086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78400f6510 0x7f7840106730 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:20.360 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 >> 172.21.15.190:0/3887290086 conn(0x7f78400f1c70 msgr2=0x7f78400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:20.360 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 shutdown_connections 2024-01-16T17:16:20.360 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:20.379+0000 7f784830b700 1 -- 172.21.15.190:0/3887290086 wait complete. 2024-01-16T17:16:20.360 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:20.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:20 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:20.379672+0000 mon.smithi154 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.190:0/3887290086' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:21.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:21.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:21.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:19.834685+0000 mgr.smithi154.sllivz (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:22.863 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.883+0000 7fa219503700 1 -- 172.21.15.190:0/3335397860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 msgr2=0x7fa2140f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:22.863 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.883+0000 7fa219503700 1 --2- 172.21.15.190:0/3335397860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 0x7fa2140f8f80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fa204009fb0 tx=0x7fa204009b30).stop 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.883+0000 7fa219503700 1 -- 172.21.15.190:0/3335397860 shutdown_connections 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.883+0000 7fa219503700 1 --2- 172.21.15.190:0/3335397860 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 0x7fa2140f8f80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.883+0000 7fa219503700 1 -- 172.21.15.190:0/3335397860 >> 172.21.15.190:0/3335397860 conn(0x7fa2140f09a0 msgr2=0x7fa2140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.884+0000 7fa219503700 1 -- 172.21.15.190:0/3335397860 shutdown_connections 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.884+0000 7fa219503700 1 -- 172.21.15.190:0/3335397860 wait complete. 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.884+0000 7fa219503700 1 Processor -- start 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.884+0000 7fa219503700 1 -- start start 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.884+0000 7fa219503700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 0x7fa214106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.884+0000 7fa219503700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2141069f0 con 0x7fa214060180 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa213fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 0x7fa214106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa213fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 0x7fa214106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33410/0 (socket says 172.21.15.190:33410) 2024-01-16T17:16:22.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa213fff700 1 -- 172.21.15.190:0/883887534 learned_addr learned my addr 172.21.15.190:0/883887534 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa213fff700 1 -- 172.21.15.190:0/883887534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa204005040 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa213fff700 1 --2- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 0x7fa214106500 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fa204013510 tx=0x7fa20400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa210ff9700 1 -- 172.21.15.190:0/883887534 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa204009320 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa210ff9700 1 -- 172.21.15.190:0/883887534 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa204013f00 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa214106bf0 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa210ff9700 1 -- 172.21.15.190:0/883887534 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa204014d90 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.885+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa214107080 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.886+0000 7fa210ff9700 1 -- 172.21.15.190:0/883887534 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2040263a0 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.886+0000 7fa210ff9700 1 --2- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa1fc02ffb0 0x7fa1fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.886+0000 7fa210ff9700 1 -- 172.21.15.190:0/883887534 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2040410e0 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.886+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa214059100 con 0x7fa214060180 2024-01-16T17:16:22.866 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.886+0000 7fa2137fe700 1 --2- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa1fc02ffb0 0x7fa1fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:22.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.887+0000 7fa2137fe700 1 --2- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa1fc02ffb0 0x7fa1fc032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa2080096a0 tx=0x7fa208006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:22.869 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:22.891+0000 7fa210ff9700 1 -- 172.21.15.190:0/883887534 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa204012920 con 0x7fa214060180 2024-01-16T17:16:23.078 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.099+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa214045ad0 con 0x7fa214060180 2024-01-16T17:16:23.078 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.100+0000 7fa210ff9700 1 -- 172.21.15.190:0/883887534 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa20402b020 con 0x7fa214060180 2024-01-16T17:16:23.079 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:23.079 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:23.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.101+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa1fc02ffb0 msgr2=0x7fa1fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:23.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.101+0000 7fa219503700 1 --2- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa1fc02ffb0 0x7fa1fc032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa2080096a0 tx=0x7fa208006b40).stop 2024-01-16T17:16:23.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.101+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 msgr2=0x7fa214106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:23.080 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.101+0000 7fa219503700 1 --2- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 0x7fa214106500 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fa204013510 tx=0x7fa20400b040).stop 2024-01-16T17:16:23.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.101+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 shutdown_connections 2024-01-16T17:16:23.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.101+0000 7fa219503700 1 --2- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa1fc02ffb0 0x7fa1fc032470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:23.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.101+0000 7fa219503700 1 --2- 172.21.15.190:0/883887534 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa214060180 0x7fa214106500 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:23.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.101+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 >> 172.21.15.190:0/883887534 conn(0x7fa2140f09a0 msgr2=0x7fa2140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:23.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.102+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 shutdown_connections 2024-01-16T17:16:23.081 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:23.102+0000 7fa219503700 1 -- 172.21.15.190:0/883887534 wait complete. 2024-01-16T17:16:23.081 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:23.828 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:21.834941+0000 mgr.smithi154.sllivz (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:23.828 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:23.100954+0000 mon.smithi154 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.190:0/883887534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:24.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:24.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:25.458 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:23.835194+0000 mgr.smithi154.sllivz (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:25.458 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:24.202197+0000 mon.smithi154 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' 2024-01-16T17:16:25.458 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:16:24.203504+0000 mgr.smithi154.sllivz (mgr.14162) 170 : cephadm [INF] Deploying daemon node-exporter.smithi154 on smithi154 2024-01-16T17:16:25.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 -- 172.21.15.190:0/2913778230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 msgr2=0x7f50a40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:25.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 --2- 172.21.15.190:0/2913778230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 0x7f50a40f66c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f50a0009fb0 tx=0x7f50a0009b30).stop 2024-01-16T17:16:25.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 -- 172.21.15.190:0/2913778230 shutdown_connections 2024-01-16T17:16:25.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 --2- 172.21.15.190:0/2913778230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 0x7f50a40f66c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:25.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 -- 172.21.15.190:0/2913778230 >> 172.21.15.190:0/2913778230 conn(0x7f50a40f1aa0 msgr2=0x7f50a40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:25.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 -- 172.21.15.190:0/2913778230 shutdown_connections 2024-01-16T17:16:25.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 -- 172.21.15.190:0/2913778230 wait complete. 2024-01-16T17:16:25.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 Processor -- start 2024-01-16T17:16:25.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.546+0000 7f50aa9e9700 1 -- start start 2024-01-16T17:16:25.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50aa9e9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 0x7f50a41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:25.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50aa9e9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50a41069a0 con 0x7f50a40f62a0 2024-01-16T17:16:25.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50a99e7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 0x7f50a41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:25.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50a99e7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 0x7f50a41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33426/0 (socket says 172.21.15.190:33426) 2024-01-16T17:16:25.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50a99e7700 1 -- 172.21.15.190:0/1611453915 learned_addr learned my addr 172.21.15.190:0/1611453915 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:25.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50a99e7700 1 -- 172.21.15.190:0/1611453915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0005040 con 0x7f50a40f62a0 2024-01-16T17:16:25.531 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50a99e7700 1 --2- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 0x7f50a41064b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f50a0013510 tx=0x7f50a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:25.531 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f509a7fc700 1 -- 172.21.15.190:0/1611453915 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50a0009320 con 0x7f50a40f62a0 2024-01-16T17:16:25.531 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a4106ba0 con 0x7f50a40f62a0 2024-01-16T17:16:25.531 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f509a7fc700 1 -- 172.21.15.190:0/1611453915 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50a0013f00 con 0x7f50a40f62a0 2024-01-16T17:16:25.531 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f509a7fc700 1 -- 172.21.15.190:0/1611453915 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50a0014dc0 con 0x7f50a40f62a0 2024-01-16T17:16:25.531 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.547+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a4107030 con 0x7f50a40f62a0 2024-01-16T17:16:25.531 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.548+0000 7f509a7fc700 1 -- 172.21.15.190:0/1611453915 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50a00095f0 con 0x7f50a40f62a0 2024-01-16T17:16:25.532 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.548+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a4045ad0 con 0x7f50a40f62a0 2024-01-16T17:16:25.532 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.548+0000 7f509a7fc700 1 --2- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f509002ff60 0x7f5090032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:25.532 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.548+0000 7f509a7fc700 1 -- 172.21.15.190:0/1611453915 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50a0041040 con 0x7f50a40f62a0 2024-01-16T17:16:25.532 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.548+0000 7f50a91e6700 1 --2- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f509002ff60 0x7f5090032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:25.532 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.549+0000 7f50a91e6700 1 --2- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f509002ff60 0x7f5090032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f50940096a0 tx=0x7f5094006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:25.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.553+0000 7f509a7fc700 1 -- 172.21.15.190:0/1611453915 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50a0014420 con 0x7f50a40f62a0 2024-01-16T17:16:25.742 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.763+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50a4107ee0 con 0x7f50a40f62a0 2024-01-16T17:16:25.742 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.764+0000 7f509a7fc700 1 -- 172.21.15.190:0/1611453915 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f50a0010230 con 0x7f50a40f62a0 2024-01-16T17:16:25.743 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:25.743 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:25.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f509002ff60 msgr2=0x7f5090032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 --2- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f509002ff60 0x7f5090032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f50940096a0 tx=0x7f5094006b40).stop 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 msgr2=0x7f50a41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 --2- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 0x7f50a41064b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f50a0013510 tx=0x7f50a000b040).stop 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 shutdown_connections 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 --2- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f509002ff60 0x7f5090032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 --2- 172.21.15.190:0/1611453915 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50a40f62a0 0x7f50a41064b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 >> 172.21.15.190:0/1611453915 conn(0x7f50a40f1aa0 msgr2=0x7f50a40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 shutdown_connections 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:25.765+0000 7f50aa9e9700 1 -- 172.21.15.190:0/1611453915 wait complete. 2024-01-16T17:16:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:26.457 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:25.764672+0000 mon.smithi154 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.190:0/1611453915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:27.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:27.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:27.457 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:25.835623+0000 mgr.smithi154.sllivz (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.294+0000 7f3e0e365700 1 -- 172.21.15.190:0/378328155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 msgr2=0x7f3e080f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.294+0000 7f3e0e365700 1 --2- 172.21.15.190:0/378328155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 0x7f3e080f6930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f3dfc009fb0 tx=0x7f3dfc009b30).stop 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.294+0000 7f3e0e365700 1 -- 172.21.15.190:0/378328155 shutdown_connections 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.294+0000 7f3e0e365700 1 --2- 172.21.15.190:0/378328155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 0x7f3e080f6930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.294+0000 7f3e0e365700 1 -- 172.21.15.190:0/378328155 >> 172.21.15.190:0/378328155 conn(0x7f3e080f1c70 msgr2=0x7f3e080f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.294+0000 7f3e0e365700 1 -- 172.21.15.190:0/378328155 shutdown_connections 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.294+0000 7f3e0e365700 1 -- 172.21.15.190:0/378328155 wait complete. 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.294+0000 7f3e0e365700 1 Processor -- start 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e0e365700 1 -- start start 2024-01-16T17:16:28.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e0e365700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 0x7f3e08106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e0e365700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08106c20 con 0x7f3e080f6510 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e07fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 0x7f3e08106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e07fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 0x7f3e08106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33454/0 (socket says 172.21.15.190:33454) 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e07fff700 1 -- 172.21.15.190:0/104408229 learned_addr learned my addr 172.21.15.190:0/104408229 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e07fff700 1 -- 172.21.15.190:0/104408229 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc005040 con 0x7f3e080f6510 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e07fff700 1 --2- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 0x7f3e08106730 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3dfc013510 tx=0x7f3dfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.295+0000 7f3e04ff9700 1 -- 172.21.15.190:0/104408229 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dfc009320 con 0x7f3e080f6510 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.296+0000 7f3e04ff9700 1 -- 172.21.15.190:0/104408229 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3dfc013f00 con 0x7f3e080f6510 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.296+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e08106e20 con 0x7f3e080f6510 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.296+0000 7f3e04ff9700 1 -- 172.21.15.190:0/104408229 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dfc014d90 con 0x7f3e080f6510 2024-01-16T17:16:28.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.296+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e081072b0 con 0x7f3e080f6510 2024-01-16T17:16:28.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.296+0000 7f3e04ff9700 1 -- 172.21.15.190:0/104408229 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3dfc014420 con 0x7f3e080f6510 2024-01-16T17:16:28.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.296+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e08045ad0 con 0x7f3e080f6510 2024-01-16T17:16:28.277 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.296+0000 7f3e04ff9700 1 --2- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3df002ffb0 0x7f3df0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:28.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.297+0000 7f3e04ff9700 1 -- 172.21.15.190:0/104408229 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3dfc041090 con 0x7f3e080f6510 2024-01-16T17:16:28.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.297+0000 7f3e077fe700 1 --2- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3df002ffb0 0x7f3df0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:28.278 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.297+0000 7f3e077fe700 1 --2- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3df002ffb0 0x7f3df0032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f3df80096a0 tx=0x7f3df8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:28.280 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.301+0000 7f3e04ff9700 1 -- 172.21.15.190:0/104408229 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dfc06d2b0 con 0x7f3e080f6510 2024-01-16T17:16:28.487 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.508+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e080fbf60 con 0x7f3e080f6510 2024-01-16T17:16:28.487 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.509+0000 7f3e04ff9700 1 -- 172.21.15.190:0/104408229 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3dfc0095f0 con 0x7f3e080f6510 2024-01-16T17:16:28.488 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:28.488 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:28.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3df002ffb0 msgr2=0x7f3df0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:28.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 --2- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3df002ffb0 0x7f3df0032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f3df80096a0 tx=0x7f3df8006b40).stop 2024-01-16T17:16:28.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 msgr2=0x7f3e08106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:28.489 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 --2- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 0x7f3e08106730 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3dfc013510 tx=0x7f3dfc00b040).stop 2024-01-16T17:16:28.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 shutdown_connections 2024-01-16T17:16:28.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 --2- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f3df002ffb0 0x7f3df0032470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:28.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 --2- 172.21.15.190:0/104408229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e080f6510 0x7f3e08106730 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:28.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 >> 172.21.15.190:0/104408229 conn(0x7f3e080f1c70 msgr2=0x7f3e080f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:28.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 shutdown_connections 2024-01-16T17:16:28.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:28.510+0000 7f3e0e365700 1 -- 172.21.15.190:0/104408229 wait complete. 2024-01-16T17:16:28.490 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:29.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:27.835868+0000 mgr.smithi154.sllivz (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:29.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:28.509752+0000 mon.smithi154 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.190:0/104408229' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:29.852 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:29.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:30.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.944+0000 7f3a00857700 1 -- 172.21.15.190:0/2272000260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 msgr2=0x7f39fc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:30.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.944+0000 7f3a00857700 1 --2- 172.21.15.190:0/2272000260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 0x7f39fc0f66c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f39ec009fb0 tx=0x7f39ec009b30).stop 2024-01-16T17:16:30.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.944+0000 7f3a00857700 1 -- 172.21.15.190:0/2272000260 shutdown_connections 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.944+0000 7f3a00857700 1 --2- 172.21.15.190:0/2272000260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 0x7f39fc0f66c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.944+0000 7f3a00857700 1 -- 172.21.15.190:0/2272000260 >> 172.21.15.190:0/2272000260 conn(0x7f39fc0f1aa0 msgr2=0x7f39fc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.944+0000 7f3a00857700 1 -- 172.21.15.190:0/2272000260 shutdown_connections 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.944+0000 7f3a00857700 1 -- 172.21.15.190:0/2272000260 wait complete. 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f3a00857700 1 Processor -- start 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f3a00857700 1 -- start start 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f3a00857700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 0x7f39fc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f3a00857700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc1069a0 con 0x7f39fc0f62a0 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f39fb7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 0x7f39fc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f39fb7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 0x7f39fc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:33464/0 (socket says 172.21.15.190:33464) 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f39fb7fe700 1 -- 172.21.15.190:0/2090121406 learned_addr learned my addr 172.21.15.190:0/2090121406 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f39fb7fe700 1 -- 172.21.15.190:0/2090121406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ec005040 con 0x7f39fc0f62a0 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f39fb7fe700 1 --2- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 0x7f39fc1064b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f39ec009830 tx=0x7f39ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.945+0000 7f39f8ff9700 1 -- 172.21.15.190:0/2090121406 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ec009320 con 0x7f39fc0f62a0 2024-01-16T17:16:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.946+0000 7f39f8ff9700 1 -- 172.21.15.190:0/2090121406 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39ec0136c0 con 0x7f39fc0f62a0 2024-01-16T17:16:30.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.946+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39fc106ba0 con 0x7f39fc0f62a0 2024-01-16T17:16:30.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.946+0000 7f39f8ff9700 1 -- 172.21.15.190:0/2090121406 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ec01ee90 con 0x7f39fc0f62a0 2024-01-16T17:16:30.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.946+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39fc107030 con 0x7f39fc0f62a0 2024-01-16T17:16:30.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.946+0000 7f39f8ff9700 1 -- 172.21.15.190:0/2090121406 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39ec014440 con 0x7f39fc0f62a0 2024-01-16T17:16:30.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.946+0000 7f39f8ff9700 1 --2- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f39e802ff10 0x7f39e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:30.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.947+0000 7f39f8ff9700 1 -- 172.21.15.190:0/2090121406 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39ec040f70 con 0x7f39fc0f62a0 2024-01-16T17:16:30.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.947+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39fc059100 con 0x7f39fc0f62a0 2024-01-16T17:16:30.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.947+0000 7f39f3fff700 1 --2- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f39e802ff10 0x7f39e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:30.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.947+0000 7f39f3fff700 1 --2- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f39e802ff10 0x7f39e80323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f39e40096a0 tx=0x7f39e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:30.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:30.952+0000 7f39f8ff9700 1 -- 172.21.15.190:0/2090121406 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39ec0273f0 con 0x7f39fc0f62a0 2024-01-16T17:16:31.136 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.157+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39fc0fbce0 con 0x7f39fc0f62a0 2024-01-16T17:16:31.137 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.158+0000 7f39f8ff9700 1 -- 172.21.15.190:0/2090121406 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f39ec01c320 con 0x7f39fc0f62a0 2024-01-16T17:16:31.137 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:31.137 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f39e802ff10 msgr2=0x7f39e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 --2- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f39e802ff10 0x7f39e80323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f39e40096a0 tx=0x7f39e4006b40).stop 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 msgr2=0x7f39fc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 --2- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 0x7f39fc1064b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f39ec009830 tx=0x7f39ec00b040).stop 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 shutdown_connections 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 --2- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f39e802ff10 0x7f39e80323d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 --2- 172.21.15.190:0/2090121406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39fc0f62a0 0x7f39fc1064b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 >> 172.21.15.190:0/2090121406 conn(0x7f39fc0f1aa0 msgr2=0x7f39fc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 shutdown_connections 2024-01-16T17:16:31.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:31.160+0000 7f3a00857700 1 -- 172.21.15.190:0/2090121406 wait complete. 2024-01-16T17:16:31.140 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:31.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:29.836216+0000 mgr.smithi154.sllivz (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:31.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:31.158995+0000 mon.smithi154 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.190:0/2090121406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:32.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:32.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:33.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.619+0000 7f1e895f4700 1 -- 172.21.15.190:0/2959923366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 msgr2=0x7f1e840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:33.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.619+0000 7f1e895f4700 1 --2- 172.21.15.190:0/2959923366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 0x7f1e840f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f1e6c009fb0 tx=0x7f1e6c009b30).stop 2024-01-16T17:16:33.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.619+0000 7f1e895f4700 1 -- 172.21.15.190:0/2959923366 shutdown_connections 2024-01-16T17:16:33.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.619+0000 7f1e895f4700 1 --2- 172.21.15.190:0/2959923366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 0x7f1e840f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:33.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.619+0000 7f1e895f4700 1 -- 172.21.15.190:0/2959923366 >> 172.21.15.190:0/2959923366 conn(0x7f1e840f0b70 msgr2=0x7f1e840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:33.600 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.620+0000 7f1e895f4700 1 -- 172.21.15.190:0/2959923366 shutdown_connections 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.620+0000 7f1e895f4700 1 -- 172.21.15.190:0/2959923366 wait complete. 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.620+0000 7f1e895f4700 1 Processor -- start 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.620+0000 7f1e895f4700 1 -- start start 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.620+0000 7f1e895f4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 0x7f1e841055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.620+0000 7f1e895f4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84105ac0 con 0x7f1e840f5410 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e82ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 0x7f1e841055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e82ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 0x7f1e841055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48112/0 (socket says 172.21.15.190:48112) 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e82ffd700 1 -- 172.21.15.190:0/985578236 learned_addr learned my addr 172.21.15.190:0/985578236 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e82ffd700 1 -- 172.21.15.190:0/985578236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c005040 con 0x7f1e840f5410 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e82ffd700 1 --2- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 0x7f1e841055d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f1e6c013510 tx=0x7f1e6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e7b7fe700 1 -- 172.21.15.190:0/985578236 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e6c009320 con 0x7f1e840f5410 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e7b7fe700 1 -- 172.21.15.190:0/985578236 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e6c013f00 con 0x7f1e840f5410 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e84105cc0 con 0x7f1e840f5410 2024-01-16T17:16:33.601 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e7b7fe700 1 -- 172.21.15.190:0/985578236 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e6c014d90 con 0x7f1e840f5410 2024-01-16T17:16:33.602 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.621+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e84106150 con 0x7f1e840f5410 2024-01-16T17:16:33.602 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.622+0000 7f1e7b7fe700 1 -- 172.21.15.190:0/985578236 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e6c0263a0 con 0x7f1e840f5410 2024-01-16T17:16:33.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.622+0000 7f1e7b7fe700 1 --2- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1e7002ff60 0x7f1e70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:33.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.622+0000 7f1e7b7fe700 1 -- 172.21.15.190:0/985578236 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e6c041140 con 0x7f1e840f5410 2024-01-16T17:16:33.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.622+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e84045ad0 con 0x7f1e840f5410 2024-01-16T17:16:33.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.622+0000 7f1e827fc700 1 --2- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1e7002ff60 0x7f1e70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:33.603 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.623+0000 7f1e827fc700 1 --2- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1e7002ff60 0x7f1e70032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1e740096a0 tx=0x7f1e74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:33.606 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.627+0000 7f1e7b7fe700 1 -- 172.21.15.190:0/985578236 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e6c012920 con 0x7f1e840f5410 2024-01-16T17:16:33.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:31.836578+0000 mgr.smithi154.sllivz (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:33.813 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.834+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e84106ea0 con 0x7f1e840f5410 2024-01-16T17:16:33.813 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.835+0000 7f1e7b7fe700 1 -- 172.21.15.190:0/985578236 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1e6c02b020 con 0x7f1e840f5410 2024-01-16T17:16:33.814 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:33.814 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:33.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1e7002ff60 msgr2=0x7f1e70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:33.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 --2- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1e7002ff60 0x7f1e70032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1e740096a0 tx=0x7f1e74006b40).stop 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 msgr2=0x7f1e841055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 --2- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 0x7f1e841055d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f1e6c013510 tx=0x7f1e6c00b040).stop 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 shutdown_connections 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 --2- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1e7002ff60 0x7f1e70032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 --2- 172.21.15.190:0/985578236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e840f5410 0x7f1e841055d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 >> 172.21.15.190:0/985578236 conn(0x7f1e840f0b70 msgr2=0x7f1e840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.836+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 shutdown_connections 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:33.837+0000 7f1e895f4700 1 -- 172.21.15.190:0/985578236 wait complete. 2024-01-16T17:16:33.816 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:34.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:34 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:33.835828+0000 mon.smithi154 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.190:0/985578236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:35.192 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:35.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:35.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:33.836824+0000 mgr.smithi154.sllivz (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:36.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.357+0000 7efee24c7700 1 -- 172.21.15.190:0/1850339762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 msgr2=0x7efedc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:36.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.357+0000 7efee24c7700 1 --2- 172.21.15.190:0/1850339762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 0x7efedc0f4be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7efec4009fb0 tx=0x7efec4009b30).stop 2024-01-16T17:16:36.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.357+0000 7efee24c7700 1 -- 172.21.15.190:0/1850339762 shutdown_connections 2024-01-16T17:16:36.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.357+0000 7efee24c7700 1 --2- 172.21.15.190:0/1850339762 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 0x7efedc0f4be0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:36.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.357+0000 7efee24c7700 1 -- 172.21.15.190:0/1850339762 >> 172.21.15.190:0/1850339762 conn(0x7efedc0f02e0 msgr2=0x7efedc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:36.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.357+0000 7efee24c7700 1 -- 172.21.15.190:0/1850339762 shutdown_connections 2024-01-16T17:16:36.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.357+0000 7efee24c7700 1 -- 172.21.15.190:0/1850339762 wait complete. 2024-01-16T17:16:36.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.358+0000 7efee24c7700 1 Processor -- start 2024-01-16T17:16:36.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.358+0000 7efee24c7700 1 -- start start 2024-01-16T17:16:36.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.358+0000 7efee24c7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 0x7efedc0f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:36.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.358+0000 7efee24c7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efedc0f57f0 con 0x7efedc0f47c0 2024-01-16T17:16:36.338 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efedbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 0x7efedc0f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efedbfff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 0x7efedc0f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48132/0 (socket says 172.21.15.190:48132) 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efedbfff700 1 -- 172.21.15.190:0/1752109088 learned_addr learned my addr 172.21.15.190:0/1752109088 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efedbfff700 1 -- 172.21.15.190:0/1752109088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec4005040 con 0x7efedc0f47c0 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efedbfff700 1 --2- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 0x7efedc0f7370 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7efec4013510 tx=0x7efec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efed8ff9700 1 -- 172.21.15.190:0/1752109088 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efec4009320 con 0x7efedc0f47c0 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efedc0f5990 con 0x7efedc0f47c0 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efed8ff9700 1 -- 172.21.15.190:0/1752109088 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efec4013f00 con 0x7efedc0f47c0 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efed8ff9700 1 -- 172.21.15.190:0/1752109088 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efec4014d90 con 0x7efedc0f47c0 2024-01-16T17:16:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.359+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efedc0f5e20 con 0x7efedc0f47c0 2024-01-16T17:16:36.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.360+0000 7efed8ff9700 1 -- 172.21.15.190:0/1752109088 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efec4014420 con 0x7efedc0f47c0 2024-01-16T17:16:36.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.360+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efedc059100 con 0x7efedc0f47c0 2024-01-16T17:16:36.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.360+0000 7efed8ff9700 1 --2- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7efec802ffb0 0x7efec8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:36.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.360+0000 7efed8ff9700 1 -- 172.21.15.190:0/1752109088 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efec40410c0 con 0x7efedc0f47c0 2024-01-16T17:16:36.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.360+0000 7efedb7fe700 1 --2- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7efec802ffb0 0x7efec8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:36.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.361+0000 7efedb7fe700 1 --2- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7efec802ffb0 0x7efec8032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7efecc0096a0 tx=0x7efecc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:36.343 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.365+0000 7efed8ff9700 1 -- 172.21.15.190:0/1752109088 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efec40095f0 con 0x7efedc0f47c0 2024-01-16T17:16:36.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.571+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efedc0fbf60 con 0x7efedc0f47c0 2024-01-16T17:16:36.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.572+0000 7efed8ff9700 1 -- 172.21.15.190:0/1752109088 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efec401db70 con 0x7efedc0f47c0 2024-01-16T17:16:36.551 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:36.551 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7efec802ffb0 msgr2=0x7efec8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 --2- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7efec802ffb0 0x7efec8032470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7efecc0096a0 tx=0x7efecc006b40).stop 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 msgr2=0x7efedc0f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 --2- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 0x7efedc0f7370 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7efec4013510 tx=0x7efec400b040).stop 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 shutdown_connections 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 --2- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7efec802ffb0 0x7efec8032470 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 --2- 172.21.15.190:0/1752109088 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efedc0f47c0 0x7efedc0f7370 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 >> 172.21.15.190:0/1752109088 conn(0x7efedc0f02e0 msgr2=0x7efedc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 shutdown_connections 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:36.573+0000 7efee24c7700 1 -- 172.21.15.190:0/1752109088 wait complete. 2024-01-16T17:16:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:37.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:35.837177+0000 mgr.smithi154.sllivz (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:37.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:36.572812+0000 mon.smithi154 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.190:0/1752109088' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:37.904 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:37.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.982+0000 7f98bdd25700 1 -- 172.21.15.190:0/890575204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 msgr2=0x7f98b80fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.982+0000 7f98bdd25700 1 --2- 172.21.15.190:0/890575204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 0x7f98b80fb570 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f98a8009fb0 tx=0x7f98a8009b30).stop 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.982+0000 7f98bdd25700 1 -- 172.21.15.190:0/890575204 shutdown_connections 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.982+0000 7f98bdd25700 1 --2- 172.21.15.190:0/890575204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 0x7f98b80fb570 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.982+0000 7f98bdd25700 1 -- 172.21.15.190:0/890575204 >> 172.21.15.190:0/890575204 conn(0x7f98b80f0b70 msgr2=0x7f98b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.982+0000 7f98bdd25700 1 -- 172.21.15.190:0/890575204 shutdown_connections 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.982+0000 7f98bdd25700 1 -- 172.21.15.190:0/890575204 wait complete. 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.982+0000 7f98bdd25700 1 Processor -- start 2024-01-16T17:16:38.964 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98bdd25700 1 -- start start 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98bdd25700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 0x7f98b81088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98bdd25700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b8108dc0 con 0x7f98b8060200 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98b77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 0x7f98b81088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98b77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 0x7f98b81088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48148/0 (socket says 172.21.15.190:48148) 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98b77fe700 1 -- 172.21.15.190:0/433194965 learned_addr learned my addr 172.21.15.190:0/433194965 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98b77fe700 1 -- 172.21.15.190:0/433194965 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a8005040 con 0x7f98b8060200 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98b77fe700 1 --2- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 0x7f98b81088d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f98a8013510 tx=0x7f98a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98affff700 1 -- 172.21.15.190:0/433194965 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a8009320 con 0x7f98b8060200 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.983+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b8108fc0 con 0x7f98b8060200 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.984+0000 7f98affff700 1 -- 172.21.15.190:0/433194965 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98a8013f00 con 0x7f98b8060200 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.984+0000 7f98affff700 1 -- 172.21.15.190:0/433194965 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98a8014dc0 con 0x7f98b8060200 2024-01-16T17:16:38.965 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.984+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b8109450 con 0x7f98b8060200 2024-01-16T17:16:38.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.984+0000 7f98affff700 1 -- 172.21.15.190:0/433194965 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98a80095f0 con 0x7f98b8060200 2024-01-16T17:16:38.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.984+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b8045ad0 con 0x7f98b8060200 2024-01-16T17:16:38.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.984+0000 7f98affff700 1 --2- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f98a402ff60 0x7f98a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:38.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.984+0000 7f98affff700 1 -- 172.21.15.190:0/433194965 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98a8041000 con 0x7f98b8060200 2024-01-16T17:16:38.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.985+0000 7f98b6ffd700 1 --2- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f98a402ff60 0x7f98a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:38.966 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.985+0000 7f98b6ffd700 1 --2- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f98a402ff60 0x7f98a4032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f98a00096a0 tx=0x7f98a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:38.968 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:38.989+0000 7f98affff700 1 -- 172.21.15.190:0/433194965 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98a801c790 con 0x7f98b8060200 2024-01-16T17:16:39.177 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.199+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98b810a300 con 0x7f98b8060200 2024-01-16T17:16:39.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.199+0000 7f98affff700 1 -- 172.21.15.190:0/433194965 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f98a8010230 con 0x7f98b8060200 2024-01-16T17:16:39.178 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:39.178 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:39.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f98a402ff60 msgr2=0x7f98a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:39.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 --2- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f98a402ff60 0x7f98a4032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f98a00096a0 tx=0x7f98a0006b40).stop 2024-01-16T17:16:39.180 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 msgr2=0x7f98b81088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:39.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 --2- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 0x7f98b81088d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f98a8013510 tx=0x7f98a800b040).stop 2024-01-16T17:16:39.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 shutdown_connections 2024-01-16T17:16:39.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 --2- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f98a402ff60 0x7f98a4032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:39.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 --2- 172.21.15.190:0/433194965 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8060200 0x7f98b81088d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:39.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 >> 172.21.15.190:0/433194965 conn(0x7f98b80f0b70 msgr2=0x7f98b80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:39.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 shutdown_connections 2024-01-16T17:16:39.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:39.201+0000 7f98bdd25700 1 -- 172.21.15.190:0/433194965 wait complete. 2024-01-16T17:16:39.181 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:39.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:37.837491+0000 mgr.smithi154.sllivz (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:39.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:39.200530+0000 mon.smithi154 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.190:0/433194965' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:40.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:40.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:41.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:39.837815+0000 mgr.smithi154.sllivz (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:41.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.801+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/3113269039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac0601e0 msgr2=0x7f0fac0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:41.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.801+0000 7f0fb0a3c700 1 --2- 172.21.15.190:0/3113269039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac0601e0 0x7f0fac0f8f20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f0f9c009fb0 tx=0x7f0f9c009b30).stop 2024-01-16T17:16:41.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.801+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/3113269039 shutdown_connections 2024-01-16T17:16:41.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.801+0000 7f0fb0a3c700 1 --2- 172.21.15.190:0/3113269039 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac0601e0 0x7f0fac0f8f20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:41.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.801+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/3113269039 >> 172.21.15.190:0/3113269039 conn(0x7f0fac0f09a0 msgr2=0x7f0fac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:41.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.801+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/3113269039 shutdown_connections 2024-01-16T17:16:41.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.801+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/3113269039 wait complete. 2024-01-16T17:16:41.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.802+0000 7f0fb0a3c700 1 Processor -- start 2024-01-16T17:16:41.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.802+0000 7f0fb0a3c700 1 -- start start 2024-01-16T17:16:41.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.802+0000 7f0fb0a3c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac1065f0 0x7f0fac106a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:41.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.802+0000 7f0fb0a3c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fac106f00 con 0x7f0fac1065f0 2024-01-16T17:16:41.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.802+0000 7f0fab7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac1065f0 0x7f0fac106a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:41.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.802+0000 7f0fab7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac1065f0 0x7f0fac106a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:48166/0 (socket says 172.21.15.190:48166) 2024-01-16T17:16:41.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.802+0000 7f0fab7fe700 1 -- 172.21.15.190:0/2216773277 learned_addr learned my addr 172.21.15.190:0/2216773277 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:41.782 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.803+0000 7f0fab7fe700 1 -- 172.21.15.190:0/2216773277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c005040 con 0x7f0fac1065f0 2024-01-16T17:16:41.783 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.803+0000 7f0fab7fe700 1 --2- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac1065f0 0x7f0fac106a10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f0f9c009f80 tx=0x7f0f9c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:41.783 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.803+0000 7f0f93fff700 1 -- 172.21.15.190:0/2216773277 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f9c0136c0 con 0x7f0fac1065f0 2024-01-16T17:16:41.783 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.803+0000 7f0f93fff700 1 -- 172.21.15.190:0/2216773277 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f9c0095f0 con 0x7f0fac1065f0 2024-01-16T17:16:41.783 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.803+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fac107100 con 0x7f0fac1065f0 2024-01-16T17:16:41.783 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.803+0000 7f0f93fff700 1 -- 172.21.15.190:0/2216773277 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f9c014420 con 0x7f0fac1065f0 2024-01-16T17:16:41.783 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.803+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fac107c50 con 0x7f0fac1065f0 2024-01-16T17:16:41.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.804+0000 7f0f93fff700 1 -- 172.21.15.190:0/2216773277 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f9c025060 con 0x7f0fac1065f0 2024-01-16T17:16:41.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.804+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fac059100 con 0x7f0fac1065f0 2024-01-16T17:16:41.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.804+0000 7f0f93fff700 1 --2- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f9402ff60 0x7f0f94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:41.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.804+0000 7f0f93fff700 1 -- 172.21.15.190:0/2216773277 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f9c040e30 con 0x7f0fac1065f0 2024-01-16T17:16:41.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.804+0000 7f0faaffd700 1 --2- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f9402ff60 0x7f0f94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:41.784 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.805+0000 7f0faaffd700 1 --2- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f9402ff60 0x7f0f94032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0fa00096a0 tx=0x7f0fa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:41.787 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:41.809+0000 7f0f93fff700 1 -- 172.21.15.190:0/2216773277 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f9c00f280 con 0x7f0fac1065f0 2024-01-16T17:16:41.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.014+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fac045ad0 con 0x7f0fac1065f0 2024-01-16T17:16:41.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.015+0000 7f0f93fff700 1 -- 172.21.15.190:0/2216773277 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f9c02d2d0 con 0x7f0fac1065f0 2024-01-16T17:16:41.994 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:41.994 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f9402ff60 msgr2=0x7f0f94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 --2- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f9402ff60 0x7f0f94032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0fa00096a0 tx=0x7f0fa0006b40).stop 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac1065f0 msgr2=0x7f0fac106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 --2- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac1065f0 0x7f0fac106a10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f0f9c009f80 tx=0x7f0f9c009b30).stop 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 shutdown_connections 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 --2- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0f9402ff60 0x7f0f94032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 --2- 172.21.15.190:0/2216773277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0fac1065f0 0x7f0fac106a10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 >> 172.21.15.190:0/2216773277 conn(0x7f0fac0f09a0 msgr2=0x7f0fac0f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 shutdown_connections 2024-01-16T17:16:41.996 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:42.017+0000 7f0fb0a3c700 1 -- 172.21.15.190:0/2216773277 wait complete. 2024-01-16T17:16:41.997 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:42.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:42 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:42.015901+0000 mon.smithi154 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.190:0/2216773277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:43.417 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:43.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:43.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:41.838226+0000 mgr.smithi154.sllivz (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:44.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.547+0000 7fb2410b6700 1 -- 172.21.15.190:0/1133205185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 msgr2=0x7fb23c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:44.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.547+0000 7fb2410b6700 1 --2- 172.21.15.190:0/1133205185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 0x7fb23c0f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fb234009960 tx=0x7fb234004d10).stop 2024-01-16T17:16:44.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.547+0000 7fb2410b6700 1 -- 172.21.15.190:0/1133205185 shutdown_connections 2024-01-16T17:16:44.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.547+0000 7fb2410b6700 1 --2- 172.21.15.190:0/1133205185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 0x7fb23c0f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:44.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.547+0000 7fb2410b6700 1 -- 172.21.15.190:0/1133205185 >> 172.21.15.190:0/1133205185 conn(0x7fb23c0f1c70 msgr2=0x7fb23c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:44.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.547+0000 7fb2410b6700 1 -- 172.21.15.190:0/1133205185 shutdown_connections 2024-01-16T17:16:44.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.547+0000 7fb2410b6700 1 -- 172.21.15.190:0/1133205185 wait complete. 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.547+0000 7fb2410b6700 1 Processor -- start 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb2410b6700 1 -- start start 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb2410b6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 0x7fb23c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb2410b6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c069e20 con 0x7fb23c0f6510 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb23ad9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 0x7fb23c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb23ad9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 0x7fb23c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:53422/0 (socket says 172.21.15.190:53422) 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb23ad9d700 1 -- 172.21.15.190:0/1317156291 learned_addr learned my addr 172.21.15.190:0/1317156291 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb23ad9d700 1 -- 172.21.15.190:0/1317156291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb234009640 con 0x7fb23c0f6510 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb23ad9d700 1 --2- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 0x7fb23c06b9a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fb234004b70 tx=0x7fb234004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb2337fe700 1 -- 172.21.15.190:0/1317156291 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb234018070 con 0x7fb23c0f6510 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c069fc0 con 0x7fb23c0f6510 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb2337fe700 1 -- 172.21.15.190:0/1317156291 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb23400b870 con 0x7fb23c0f6510 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb2337fe700 1 -- 172.21.15.190:0/1317156291 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb23401c560 con 0x7fb23c0f6510 2024-01-16T17:16:44.528 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.548+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c06a450 con 0x7fb23c0f6510 2024-01-16T17:16:44.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.549+0000 7fb2337fe700 1 -- 172.21.15.190:0/1317156291 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb23401c6c0 con 0x7fb23c0f6510 2024-01-16T17:16:44.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.549+0000 7fb2337fe700 1 --2- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22802ff60 0x7fb228032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:44.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.549+0000 7fb2337fe700 1 -- 172.21.15.190:0/1317156291 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2340412b0 con 0x7fb23c0f6510 2024-01-16T17:16:44.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.549+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c059100 con 0x7fb23c0f6510 2024-01-16T17:16:44.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.550+0000 7fb23a59c700 1 --2- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22802ff60 0x7fb228032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:44.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.550+0000 7fb23a59c700 1 --2- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22802ff60 0x7fb228032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb2240096a0 tx=0x7fb224006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:44.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.554+0000 7fb2337fe700 1 -- 172.21.15.190:0/1317156291 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb234012be0 con 0x7fb23c0f6510 2024-01-16T17:16:44.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.758+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb23c0fbf60 con 0x7fb23c0f6510 2024-01-16T17:16:44.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.758+0000 7fb2337fe700 1 -- 172.21.15.190:0/1317156291 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb234004e10 con 0x7fb23c0f6510 2024-01-16T17:16:44.737 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:44.737 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22802ff60 msgr2=0x7fb228032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 --2- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22802ff60 0x7fb228032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fb2240096a0 tx=0x7fb224006b40).stop 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 msgr2=0x7fb23c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 --2- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 0x7fb23c06b9a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fb234004b70 tx=0x7fb234004d10).stop 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 shutdown_connections 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 --2- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb22802ff60 0x7fb228032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 --2- 172.21.15.190:0/1317156291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb23c0f6510 0x7fb23c06b9a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 >> 172.21.15.190:0/1317156291 conn(0x7fb23c0f1c70 msgr2=0x7fb23c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 shutdown_connections 2024-01-16T17:16:44.739 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:44.760+0000 7fb2410b6700 1 -- 172.21.15.190:0/1317156291 wait complete. 2024-01-16T17:16:44.740 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:45.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:43.838576+0000 mgr.smithi154.sllivz (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:45.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:44.759421+0000 mon.smithi154 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.190:0/1317156291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:46.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:46.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:47.230 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.249+0000 7f43e4286700 1 -- 172.21.15.190:0/4250702003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 msgr2=0x7f43dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:47.230 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.249+0000 7f43e4286700 1 --2- 172.21.15.190:0/4250702003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 0x7f43dc0f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f43d0009fb0 tx=0x7f43d0009b30).stop 2024-01-16T17:16:47.230 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.249+0000 7f43e4286700 1 -- 172.21.15.190:0/4250702003 shutdown_connections 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.249+0000 7f43e4286700 1 --2- 172.21.15.190:0/4250702003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 0x7f43dc0f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.249+0000 7f43e4286700 1 -- 172.21.15.190:0/4250702003 >> 172.21.15.190:0/4250702003 conn(0x7f43dc0f0b70 msgr2=0x7f43dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.249+0000 7f43e4286700 1 -- 172.21.15.190:0/4250702003 shutdown_connections 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.249+0000 7f43e4286700 1 -- 172.21.15.190:0/4250702003 wait complete. 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e4286700 1 Processor -- start 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e4286700 1 -- start start 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e4286700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 0x7f43dc106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e4286700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc106c10 con 0x7f43dc0f5410 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e2022700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 0x7f43dc106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e2022700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 0x7f43dc106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:53432/0 (socket says 172.21.15.190:53432) 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e2022700 1 -- 172.21.15.190:0/521088716 learned_addr learned my addr 172.21.15.190:0/521088716 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e2022700 1 -- 172.21.15.190:0/521088716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d0005040 con 0x7f43dc0f5410 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.250+0000 7f43e2022700 1 --2- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 0x7f43dc106720 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f43d0013510 tx=0x7f43d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.251+0000 7f43ceffd700 1 -- 172.21.15.190:0/521088716 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43d0009320 con 0x7f43dc0f5410 2024-01-16T17:16:47.231 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.251+0000 7f43ceffd700 1 -- 172.21.15.190:0/521088716 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43d0013f00 con 0x7f43dc0f5410 2024-01-16T17:16:47.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.251+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43dc106e10 con 0x7f43dc0f5410 2024-01-16T17:16:47.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.251+0000 7f43ceffd700 1 -- 172.21.15.190:0/521088716 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43d0014d90 con 0x7f43dc0f5410 2024-01-16T17:16:47.232 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.251+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43dc1072a0 con 0x7f43dc0f5410 2024-01-16T17:16:47.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.251+0000 7f43ceffd700 1 -- 172.21.15.190:0/521088716 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43d00095f0 con 0x7f43dc0f5410 2024-01-16T17:16:47.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.251+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43dc045ad0 con 0x7f43dc0f5410 2024-01-16T17:16:47.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.252+0000 7f43ceffd700 1 --2- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f43c802ff60 0x7f43c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:47.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.252+0000 7f43ceffd700 1 -- 172.21.15.190:0/521088716 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43d00410c0 con 0x7f43dc0f5410 2024-01-16T17:16:47.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.252+0000 7f43e1821700 1 --2- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f43c802ff60 0x7f43c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:47.233 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.252+0000 7f43e1821700 1 --2- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f43c802ff60 0x7f43c8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f43d80096a0 tx=0x7f43d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:47.235 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.256+0000 7f43ceffd700 1 -- 172.21.15.190:0/521088716 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43d0014420 con 0x7f43dc0f5410 2024-01-16T17:16:47.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.459+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43dc108150 con 0x7f43dc0f5410 2024-01-16T17:16:47.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.460+0000 7f43ceffd700 1 -- 172.21.15.190:0/521088716 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f43d0010230 con 0x7f43dc0f5410 2024-01-16T17:16:47.440 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:47.440 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:47.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.461+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f43c802ff60 msgr2=0x7f43c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:47.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.461+0000 7f43e4286700 1 --2- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f43c802ff60 0x7f43c8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f43d80096a0 tx=0x7f43d8006b40).stop 2024-01-16T17:16:47.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.461+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 msgr2=0x7f43dc106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:47.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.461+0000 7f43e4286700 1 --2- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 0x7f43dc106720 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f43d0013510 tx=0x7f43d000b040).stop 2024-01-16T17:16:47.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.461+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 shutdown_connections 2024-01-16T17:16:47.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.461+0000 7f43e4286700 1 --2- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f43c802ff60 0x7f43c8032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:47.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.461+0000 7f43e4286700 1 --2- 172.21.15.190:0/521088716 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43dc0f5410 0x7f43dc106720 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:47.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.461+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 >> 172.21.15.190:0/521088716 conn(0x7f43dc0f0b70 msgr2=0x7f43dc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:47.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.462+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 shutdown_connections 2024-01-16T17:16:47.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:47.462+0000 7f43e4286700 1 -- 172.21.15.190:0/521088716 wait complete. 2024-01-16T17:16:47.442 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:47.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:45.838935+0000 mgr.smithi154.sllivz (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:48.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:48 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:47.461975+0000 mon.smithi154 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.190:0/521088716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:48.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:48.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:49.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:47.839314+0000 mgr.smithi154.sllivz (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:49.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.939+0000 7f64b56d4700 1 -- 172.21.15.190:0/2686593220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 msgr2=0x7f64b00f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:49.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.939+0000 7f64b56d4700 1 --2- 172.21.15.190:0/2686593220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 0x7f64b00f5df0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f64a4009fb0 tx=0x7f64a4009b30).stop 2024-01-16T17:16:49.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.939+0000 7f64b56d4700 1 -- 172.21.15.190:0/2686593220 shutdown_connections 2024-01-16T17:16:49.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.939+0000 7f64b56d4700 1 --2- 172.21.15.190:0/2686593220 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 0x7f64b00f5df0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:49.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.939+0000 7f64b56d4700 1 -- 172.21.15.190:0/2686593220 >> 172.21.15.190:0/2686593220 conn(0x7f64b00f02e0 msgr2=0x7f64b00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:49.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.939+0000 7f64b56d4700 1 -- 172.21.15.190:0/2686593220 shutdown_connections 2024-01-16T17:16:49.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.939+0000 7f64b56d4700 1 -- 172.21.15.190:0/2686593220 wait complete. 2024-01-16T17:16:49.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.940+0000 7f64b56d4700 1 Processor -- start 2024-01-16T17:16:49.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.940+0000 7f64b56d4700 1 -- start start 2024-01-16T17:16:49.924 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.940+0000 7f64b56d4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 0x7f64b01093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:49.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.940+0000 7f64b56d4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b00f53b0 con 0x7f64b00f59d0 2024-01-16T17:16:49.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.940+0000 7f64aeffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 0x7f64b01093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:49.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.940+0000 7f64aeffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 0x7f64b01093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:53454/0 (socket says 172.21.15.190:53454) 2024-01-16T17:16:49.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.940+0000 7f64aeffd700 1 -- 172.21.15.190:0/3029933755 learned_addr learned my addr 172.21.15.190:0/3029933755 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:49.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.941+0000 7f64aeffd700 1 -- 172.21.15.190:0/3029933755 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a4005040 con 0x7f64b00f59d0 2024-01-16T17:16:49.925 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.941+0000 7f64aeffd700 1 --2- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 0x7f64b01093c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f64a4013510 tx=0x7f64a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:49.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.941+0000 7f64977fe700 1 -- 172.21.15.190:0/3029933755 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a4009320 con 0x7f64b00f59d0 2024-01-16T17:16:49.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.941+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b00f55b0 con 0x7f64b00f59d0 2024-01-16T17:16:49.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.941+0000 7f64977fe700 1 -- 172.21.15.190:0/3029933755 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64a4013f00 con 0x7f64b00f59d0 2024-01-16T17:16:49.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.941+0000 7f64977fe700 1 -- 172.21.15.190:0/3029933755 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64a4014d90 con 0x7f64b00f59d0 2024-01-16T17:16:49.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.941+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b0109a70 con 0x7f64b00f59d0 2024-01-16T17:16:49.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.942+0000 7f64977fe700 1 -- 172.21.15.190:0/3029933755 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64a40263a0 con 0x7f64b00f59d0 2024-01-16T17:16:49.926 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.942+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b0045ad0 con 0x7f64b00f59d0 2024-01-16T17:16:49.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.942+0000 7f64977fe700 1 --2- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:49.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.942+0000 7f64977fe700 1 -- 172.21.15.190:0/3029933755 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64a4041360 con 0x7f64b00f59d0 2024-01-16T17:16:49.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.942+0000 7f64ae7fc700 1 --2- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:49.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.943+0000 7f64ae7fc700 1 --2- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f649802ff60 0x7f6498032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f64a00096a0 tx=0x7f64a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:49.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:49.947+0000 7f64977fe700 1 -- 172.21.15.190:0/3029933755 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64a40126f0 con 0x7f64b00f59d0 2024-01-16T17:16:50.131 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.151+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64b010a290 con 0x7f64b00f59d0 2024-01-16T17:16:50.131 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.152+0000 7f64977fe700 1 -- 172.21.15.190:0/3029933755 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f64a4012330 con 0x7f64b00f59d0 2024-01-16T17:16:50.132 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:50.132 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:50.133 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f649802ff60 msgr2=0x7f6498032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 --2- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f649802ff60 0x7f6498032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f64a00096a0 tx=0x7f64a0006b40).stop 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 msgr2=0x7f64b01093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 --2- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 0x7f64b01093c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f64a4013510 tx=0x7f64a400b040).stop 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 shutdown_connections 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 --2- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f649802ff60 0x7f6498032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 --2- 172.21.15.190:0/3029933755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64b00f59d0 0x7f64b01093c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 >> 172.21.15.190:0/3029933755 conn(0x7f64b00f02e0 msgr2=0x7f64b00f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 shutdown_connections 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:50.153+0000 7f64b56d4700 1 -- 172.21.15.190:0/3029933755 wait complete. 2024-01-16T17:16:50.134 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:50.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:50 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:50.153664+0000 mon.smithi154 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.190:0/3029933755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:51.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:51.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:51.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:49.839635+0000 mgr.smithi154.sllivz (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.640+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/1018181732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 msgr2=0x7f6c940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.640+0000 7f6c9bc1b700 1 --2- 172.21.15.190:0/1018181732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 0x7f6c940f6930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f6c84009fb0 tx=0x7f6c84009b30).stop 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.640+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/1018181732 shutdown_connections 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.640+0000 7f6c9bc1b700 1 --2- 172.21.15.190:0/1018181732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 0x7f6c940f6930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.640+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/1018181732 >> 172.21.15.190:0/1018181732 conn(0x7f6c940f1c70 msgr2=0x7f6c940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.640+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/1018181732 shutdown_connections 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.640+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/1018181732 wait complete. 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.641+0000 7f6c9bc1b700 1 Processor -- start 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.641+0000 7f6c9bc1b700 1 -- start start 2024-01-16T17:16:52.622 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.641+0000 7f6c9bc1b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 0x7f6c94106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.641+0000 7f6c9bc1b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c94106c20 con 0x7f6c940f6510 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.641+0000 7f6c999b7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 0x7f6c94106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.641+0000 7f6c999b7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 0x7f6c94106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60118/0 (socket says 172.21.15.190:60118) 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.641+0000 7f6c999b7700 1 -- 172.21.15.190:0/2420912103 learned_addr learned my addr 172.21.15.190:0/2420912103 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.642+0000 7f6c999b7700 1 -- 172.21.15.190:0/2420912103 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c84005040 con 0x7f6c940f6510 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.642+0000 7f6c999b7700 1 --2- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 0x7f6c94106730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6c84013510 tx=0x7f6c8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.642+0000 7f6c8a7fc700 1 -- 172.21.15.190:0/2420912103 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c84009320 con 0x7f6c940f6510 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.642+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c94106e20 con 0x7f6c940f6510 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.642+0000 7f6c8a7fc700 1 -- 172.21.15.190:0/2420912103 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c84013f00 con 0x7f6c940f6510 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.642+0000 7f6c8a7fc700 1 -- 172.21.15.190:0/2420912103 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c84014d90 con 0x7f6c940f6510 2024-01-16T17:16:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.642+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c941072b0 con 0x7f6c940f6510 2024-01-16T17:16:52.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.643+0000 7f6c8a7fc700 1 -- 172.21.15.190:0/2420912103 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c840095f0 con 0x7f6c940f6510 2024-01-16T17:16:52.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.643+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c94045ad0 con 0x7f6c940f6510 2024-01-16T17:16:52.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.643+0000 7f6c8a7fc700 1 --2- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c8002ff60 0x7f6c80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:52.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.643+0000 7f6c8a7fc700 1 -- 172.21.15.190:0/2420912103 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c84040fc0 con 0x7f6c940f6510 2024-01-16T17:16:52.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.643+0000 7f6c991b6700 1 --2- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c8002ff60 0x7f6c80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:52.624 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.643+0000 7f6c991b6700 1 --2- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c8002ff60 0x7f6c80032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f6c900096a0 tx=0x7f6c90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:52.627 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.647+0000 7f6c8a7fc700 1 -- 172.21.15.190:0/2420912103 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c84014420 con 0x7f6c940f6510 2024-01-16T17:16:52.837 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.857+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c940fbf60 con 0x7f6c940f6510 2024-01-16T17:16:52.837 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.858+0000 7f6c8a7fc700 1 -- 172.21.15.190:0/2420912103 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6c84010230 con 0x7f6c940f6510 2024-01-16T17:16:52.838 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:52.838 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:52.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.859+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c8002ff60 msgr2=0x7f6c80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:52.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.859+0000 7f6c9bc1b700 1 --2- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c8002ff60 0x7f6c80032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f6c900096a0 tx=0x7f6c90006b40).stop 2024-01-16T17:16:52.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.859+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 msgr2=0x7f6c94106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:52.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.859+0000 7f6c9bc1b700 1 --2- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 0x7f6c94106730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6c84013510 tx=0x7f6c8400b040).stop 2024-01-16T17:16:52.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.860+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 shutdown_connections 2024-01-16T17:16:52.839 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.860+0000 7f6c9bc1b700 1 --2- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f6c8002ff60 0x7f6c80032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:52.840 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.860+0000 7f6c9bc1b700 1 --2- 172.21.15.190:0/2420912103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c940f6510 0x7f6c94106730 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:52.840 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.860+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 >> 172.21.15.190:0/2420912103 conn(0x7f6c940f1c70 msgr2=0x7f6c940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:52.840 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.860+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 shutdown_connections 2024-01-16T17:16:52.840 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:52.860+0000 7f6c9bc1b700 1 -- 172.21.15.190:0/2420912103 wait complete. 2024-01-16T17:16:52.840 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:53.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:51.839995+0000 mgr.smithi154.sllivz (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:53.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:52.859707+0000 mon.smithi154 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.190:0/2420912103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:54.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:54.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.380+0000 7f200cba6700 1 -- 172.21.15.190:0/3050109838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 msgr2=0x7f20080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.380+0000 7f200cba6700 1 --2- 172.21.15.190:0/3050109838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 0x7f20080f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f1ff8009fb0 tx=0x7f1ff8009b30).stop 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.380+0000 7f200cba6700 1 -- 172.21.15.190:0/3050109838 shutdown_connections 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.380+0000 7f200cba6700 1 --2- 172.21.15.190:0/3050109838 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 0x7f20080f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.380+0000 7f200cba6700 1 -- 172.21.15.190:0/3050109838 >> 172.21.15.190:0/3050109838 conn(0x7f20080f0b70 msgr2=0x7f20080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.380+0000 7f200cba6700 1 -- 172.21.15.190:0/3050109838 shutdown_connections 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.380+0000 7f200cba6700 1 -- 172.21.15.190:0/3050109838 wait complete. 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200cba6700 1 Processor -- start 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200cba6700 1 -- start start 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200cba6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 0x7f2008105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:55.362 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200cba6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2008105b20 con 0x7f20080f5410 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 0x7f2008105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 0x7f2008105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60134/0 (socket says 172.21.15.190:60134) 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200659c700 1 -- 172.21.15.190:0/3876136554 learned_addr learned my addr 172.21.15.190:0/3876136554 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200659c700 1 -- 172.21.15.190:0/3876136554 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff8005040 con 0x7f20080f5410 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.381+0000 7f200659c700 1 --2- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 0x7f2008105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f1ff8013510 tx=0x7f1ff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.382+0000 7f1ffeffd700 1 -- 172.21.15.190:0/3876136554 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ff8009320 con 0x7f20080f5410 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.382+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2008105d20 con 0x7f20080f5410 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.382+0000 7f1ffeffd700 1 -- 172.21.15.190:0/3876136554 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ff8013f00 con 0x7f20080f5410 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.382+0000 7f1ffeffd700 1 -- 172.21.15.190:0/3876136554 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ff8014d90 con 0x7f20080f5410 2024-01-16T17:16:55.363 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.382+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20081061b0 con 0x7f20080f5410 2024-01-16T17:16:55.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.382+0000 7f1ffeffd700 1 -- 172.21.15.190:0/3876136554 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ff80095f0 con 0x7f20080f5410 2024-01-16T17:16:55.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.383+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2008045ad0 con 0x7f20080f5410 2024-01-16T17:16:55.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.383+0000 7f1ffeffd700 1 --2- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ff402ff60 0x7f1ff4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:55.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.383+0000 7f1ffeffd700 1 -- 172.21.15.190:0/3876136554 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ff8040fd0 con 0x7f20080f5410 2024-01-16T17:16:55.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.383+0000 7f2005d9b700 1 --2- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ff402ff60 0x7f1ff4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:55.364 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.383+0000 7f2005d9b700 1 --2- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ff402ff60 0x7f1ff4032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1ff00096a0 tx=0x7f1ff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:55.367 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.388+0000 7f1ffeffd700 1 -- 172.21.15.190:0/3876136554 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ff8014420 con 0x7f20080f5410 2024-01-16T17:16:55.576 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.597+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20080fae60 con 0x7f20080f5410 2024-01-16T17:16:55.577 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.597+0000 7f1ffeffd700 1 -- 172.21.15.190:0/3876136554 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1ff801d7f0 con 0x7f20080f5410 2024-01-16T17:16:55.577 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:55.577 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:55.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ff402ff60 msgr2=0x7f1ff4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:55.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 --2- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ff402ff60 0x7f1ff4032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f1ff00096a0 tx=0x7f1ff0006b40).stop 2024-01-16T17:16:55.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 msgr2=0x7f2008105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:55.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 --2- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 0x7f2008105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f1ff8013510 tx=0x7f1ff800b040).stop 2024-01-16T17:16:55.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 shutdown_connections 2024-01-16T17:16:55.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 --2- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ff402ff60 0x7f1ff4032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:55.579 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 --2- 172.21.15.190:0/3876136554 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20080f5410 0x7f2008105630 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:55.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 >> 172.21.15.190:0/3876136554 conn(0x7f20080f0b70 msgr2=0x7f20080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:55.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 shutdown_connections 2024-01-16T17:16:55.580 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:55.599+0000 7f200cba6700 1 -- 172.21.15.190:0/3876136554 wait complete. 2024-01-16T17:16:55.580 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:55.706 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:53.840320+0000 mgr.smithi154.sllivz (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:56.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:55.599512+0000 mon.smithi154 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.190:0/3876136554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:56.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:55.930719+0000 mon.smithi154 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:16:56.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:55.931839+0000 mon.smithi154 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:16:56.948 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:56.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:57.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:55.840560+0000 mgr.smithi154.sllivz (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:16:58.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 -- 172.21.15.190:0/959312417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 msgr2=0x7f057c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:58.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 --2- 172.21.15.190:0/959312417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 0x7f057c0f90f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f0570009fb0 tx=0x7f0570009b30).stop 2024-01-16T17:16:58.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 -- 172.21.15.190:0/959312417 shutdown_connections 2024-01-16T17:16:58.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 --2- 172.21.15.190:0/959312417 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 0x7f057c0f90f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:58.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 -- 172.21.15.190:0/959312417 >> 172.21.15.190:0/959312417 conn(0x7f057c0f0b70 msgr2=0x7f057c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:58.055 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 -- 172.21.15.190:0/959312417 shutdown_connections 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 -- 172.21.15.190:0/959312417 wait complete. 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 Processor -- start 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.075+0000 7f0582bce700 1 -- start start 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.076+0000 7f0582bce700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 0x7f057c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.076+0000 7f0582bce700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057c106bd0 con 0x7f057c060280 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.076+0000 7f058096a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 0x7f057c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.076+0000 7f058096a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 0x7f057c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60154/0 (socket says 172.21.15.190:60154) 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.076+0000 7f058096a700 1 -- 172.21.15.190:0/3578062321 learned_addr learned my addr 172.21.15.190:0/3578062321 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.076+0000 7f058096a700 1 -- 172.21.15.190:0/3578062321 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570005040 con 0x7f057c060280 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.076+0000 7f058096a700 1 --2- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 0x7f057c1066e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f0570013510 tx=0x7f057000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.077+0000 7f05797fa700 1 -- 172.21.15.190:0/3578062321 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0570009320 con 0x7f057c060280 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.077+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057c106dd0 con 0x7f057c060280 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.077+0000 7f05797fa700 1 -- 172.21.15.190:0/3578062321 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0570013f00 con 0x7f057c060280 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.077+0000 7f05797fa700 1 -- 172.21.15.190:0/3578062321 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0570014dc0 con 0x7f057c060280 2024-01-16T17:16:58.056 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.077+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057c107260 con 0x7f057c060280 2024-01-16T17:16:58.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.077+0000 7f05797fa700 1 -- 172.21.15.190:0/3578062321 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0570014420 con 0x7f057c060280 2024-01-16T17:16:58.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.077+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057c045ad0 con 0x7f057c060280 2024-01-16T17:16:58.057 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.077+0000 7f05797fa700 1 --2- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f056c02ff60 0x7f056c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:16:58.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.078+0000 7f05797fa700 1 -- 172.21.15.190:0/3578062321 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0570041110 con 0x7f057c060280 2024-01-16T17:16:58.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.078+0000 7f057bfff700 1 --2- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f056c02ff60 0x7f056c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:16:58.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.078+0000 7f057bfff700 1 --2- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f056c02ff60 0x7f056c032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f05680096a0 tx=0x7f0568006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:16:58.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.082+0000 7f05797fa700 1 -- 172.21.15.190:0/3578062321 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f05700095f0 con 0x7f057c060280 2024-01-16T17:16:58.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.289+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f057c108110 con 0x7f057c060280 2024-01-16T17:16:58.268 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.290+0000 7f05797fa700 1 -- 172.21.15.190:0/3578062321 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f057001cb10 con 0x7f057c060280 2024-01-16T17:16:58.268 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:16:58.268 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:16:58.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.291+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f056c02ff60 msgr2=0x7f056c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:58.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.291+0000 7f0582bce700 1 --2- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f056c02ff60 0x7f056c032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f05680096a0 tx=0x7f0568006b40).stop 2024-01-16T17:16:58.270 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.291+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 msgr2=0x7f057c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:16:58.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.291+0000 7f0582bce700 1 --2- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 0x7f057c1066e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f0570013510 tx=0x7f057000b040).stop 2024-01-16T17:16:58.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.291+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 shutdown_connections 2024-01-16T17:16:58.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.291+0000 7f0582bce700 1 --2- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f056c02ff60 0x7f056c032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:58.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.292+0000 7f0582bce700 1 --2- 172.21.15.190:0/3578062321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f057c060280 0x7f057c1066e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:16:58.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.292+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 >> 172.21.15.190:0/3578062321 conn(0x7f057c0f0b70 msgr2=0x7f057c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:16:58.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.292+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 shutdown_connections 2024-01-16T17:16:58.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:16:58.292+0000 7f0582bce700 1 -- 172.21.15.190:0/3578062321 wait complete. 2024-01-16T17:16:58.271 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:16:58.706 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:58 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:16:58.290822+0000 mon.smithi154 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.190:0/3578062321' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:16:59.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:16:59.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:16:59.706 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:16:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:57.840822+0000 mgr.smithi154.sllivz (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:00.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.873+0000 7f08111ee700 1 -- 172.21.15.190:0/161705690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 msgr2=0x7f080c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:00.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.873+0000 7f08111ee700 1 --2- 172.21.15.190:0/161705690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 0x7f080c0f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f0800009fb0 tx=0x7f0800009b30).stop 2024-01-16T17:17:00.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.873+0000 7f08111ee700 1 -- 172.21.15.190:0/161705690 shutdown_connections 2024-01-16T17:17:00.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.873+0000 7f08111ee700 1 --2- 172.21.15.190:0/161705690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 0x7f080c0f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:00.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.873+0000 7f08111ee700 1 -- 172.21.15.190:0/161705690 >> 172.21.15.190:0/161705690 conn(0x7f080c0f0b70 msgr2=0x7f080c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:00.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.873+0000 7f08111ee700 1 -- 172.21.15.190:0/161705690 shutdown_connections 2024-01-16T17:17:00.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.873+0000 7f08111ee700 1 -- 172.21.15.190:0/161705690 wait complete. 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.874+0000 7f08111ee700 1 Processor -- start 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.874+0000 7f08111ee700 1 -- start start 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.874+0000 7f08111ee700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 0x7f080c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.874+0000 7f08111ee700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c060280 con 0x7f080c0f5410 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.874+0000 7f080ad9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 0x7f080c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.874+0000 7f080ad9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 0x7f080c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60164/0 (socket says 172.21.15.190:60164) 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.874+0000 7f080ad9d700 1 -- 172.21.15.190:0/365825940 learned_addr learned my addr 172.21.15.190:0/365825940 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.875+0000 7f080ad9d700 1 -- 172.21.15.190:0/365825940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0800005040 con 0x7f080c0f5410 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.875+0000 7f080ad9d700 1 --2- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 0x7f080c1093f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f0800013510 tx=0x7f080000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.875+0000 7f07f37fe700 1 -- 172.21.15.190:0/365825940 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0800009320 con 0x7f080c0f5410 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.875+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c060480 con 0x7f080c0f5410 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.875+0000 7f07f37fe700 1 -- 172.21.15.190:0/365825940 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0800013f00 con 0x7f080c0f5410 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.875+0000 7f07f37fe700 1 -- 172.21.15.190:0/365825940 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0800014d90 con 0x7f080c0f5410 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.875+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c109b20 con 0x7f080c0f5410 2024-01-16T17:17:00.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.876+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c045ad0 con 0x7f080c0f5410 2024-01-16T17:17:00.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.876+0000 7f07f37fe700 1 -- 172.21.15.190:0/365825940 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08000095f0 con 0x7f080c0f5410 2024-01-16T17:17:00.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.876+0000 7f07f37fe700 1 --2- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f07f402ff60 0x7f07f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:00.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.876+0000 7f07f37fe700 1 -- 172.21.15.190:0/365825940 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0800041000 con 0x7f080c0f5410 2024-01-16T17:17:00.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.876+0000 7f080a59c700 1 --2- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f07f402ff60 0x7f07f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:00.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.877+0000 7f080a59c700 1 --2- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f07f402ff60 0x7f07f4032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f07fc0096a0 tx=0x7f07fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:00.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:00.881+0000 7f07f37fe700 1 -- 172.21.15.190:0/365825940 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0800014420 con 0x7f080c0f5410 2024-01-16T17:17:01.062 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.084+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f080c10a340 con 0x7f080c0f5410 2024-01-16T17:17:01.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.085+0000 7f07f37fe700 1 -- 172.21.15.190:0/365825940 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0800010230 con 0x7f080c0f5410 2024-01-16T17:17:01.063 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:01.064 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:01.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f07f402ff60 msgr2=0x7f07f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:01.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 --2- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f07f402ff60 0x7f07f4032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f07fc0096a0 tx=0x7f07fc006b40).stop 2024-01-16T17:17:01.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 msgr2=0x7f080c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:01.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 --2- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 0x7f080c1093f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f0800013510 tx=0x7f080000b040).stop 2024-01-16T17:17:01.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 shutdown_connections 2024-01-16T17:17:01.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 --2- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f07f402ff60 0x7f07f4032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:01.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 --2- 172.21.15.190:0/365825940 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f080c0f5410 0x7f080c1093f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:01.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 >> 172.21.15.190:0/365825940 conn(0x7f080c0f0b70 msgr2=0x7f080c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:01.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 shutdown_connections 2024-01-16T17:17:01.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:01.086+0000 7f08111ee700 1 -- 172.21.15.190:0/365825940 wait complete. 2024-01-16T17:17:01.066 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:01.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:16:59.841142+0000 mgr.smithi154.sllivz (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:01.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:01.085702+0000 mon.smithi154 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.190:0/365825940' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:02.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:02.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.566+0000 7f1cfefb3700 1 -- 172.21.15.190:0/232998076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 msgr2=0x7f1cf80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.566+0000 7f1cfefb3700 1 --2- 172.21.15.190:0/232998076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 0x7f1cf80f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f1cec009fb0 tx=0x7f1cec009b30).stop 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.566+0000 7f1cfefb3700 1 -- 172.21.15.190:0/232998076 shutdown_connections 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.566+0000 7f1cfefb3700 1 --2- 172.21.15.190:0/232998076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 0x7f1cf80f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.566+0000 7f1cfefb3700 1 -- 172.21.15.190:0/232998076 >> 172.21.15.190:0/232998076 conn(0x7f1cf80f0b70 msgr2=0x7f1cf80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.566+0000 7f1cfefb3700 1 -- 172.21.15.190:0/232998076 shutdown_connections 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.566+0000 7f1cfefb3700 1 -- 172.21.15.190:0/232998076 wait complete. 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.567+0000 7f1cfefb3700 1 Processor -- start 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.567+0000 7f1cfefb3700 1 -- start start 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.567+0000 7f1cfefb3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 0x7f1cf810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:03.546 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.567+0000 7f1cfefb3700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf8107240 con 0x7f1cf80f5410 2024-01-16T17:17:03.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cfcd4f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 0x7f1cf810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:03.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cfcd4f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 0x7f1cf810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60494/0 (socket says 172.21.15.190:60494) 2024-01-16T17:17:03.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cfcd4f700 1 -- 172.21.15.190:0/3143773581 learned_addr learned my addr 172.21.15.190:0/3143773581 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cfcd4f700 1 -- 172.21.15.190:0/3143773581 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec005040 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cfcd4f700 1 --2- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 0x7f1cf810b850 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f1cec013510 tx=0x7f1cec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cf57fa700 1 -- 172.21.15.190:0/3143773581 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cec009320 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf8060220 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cf57fa700 1 -- 172.21.15.190:0/3143773581 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cec013f00 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf810bd40 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.568+0000 7f1cf57fa700 1 -- 172.21.15.190:0/3143773581 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cec014db0 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.569+0000 7f1cf57fa700 1 -- 172.21.15.190:0/3143773581 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1cec0095f0 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.569+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf8045ad0 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.569+0000 7f1cf57fa700 1 --2- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ce802ff60 0x7f1ce8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.569+0000 7f1cf57fa700 1 -- 172.21.15.190:0/3143773581 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1cec041060 con 0x7f1cf80f5410 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.569+0000 7f1cf7fff700 1 --2- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ce802ff60 0x7f1ce8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:03.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.570+0000 7f1cf7fff700 1 --2- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ce802ff60 0x7f1ce8032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1ce40096a0 tx=0x7f1ce4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:03.552 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.574+0000 7f1cf57fa700 1 -- 172.21.15.190:0/3143773581 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cec014420 con 0x7f1cf80f5410 2024-01-16T17:17:03.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:01.841481+0000 mgr.smithi154.sllivz (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:03.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.781+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cf80fae60 con 0x7f1cf80f5410 2024-01-16T17:17:03.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.782+0000 7f1cf57fa700 1 -- 172.21.15.190:0/3143773581 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1cec01c920 con 0x7f1cf80f5410 2024-01-16T17:17:03.760 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:03.760 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ce802ff60 msgr2=0x7f1ce8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 --2- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ce802ff60 0x7f1ce8032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f1ce40096a0 tx=0x7f1ce4006b40).stop 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 msgr2=0x7f1cf810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 --2- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 0x7f1cf810b850 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f1cec013510 tx=0x7f1cec00b040).stop 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 shutdown_connections 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 --2- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1ce802ff60 0x7f1ce8032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 --2- 172.21.15.190:0/3143773581 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1cf80f5410 0x7f1cf810b850 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 >> 172.21.15.190:0/3143773581 conn(0x7f1cf80f0b70 msgr2=0x7f1cf80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:03.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 shutdown_connections 2024-01-16T17:17:03.763 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:03.783+0000 7f1cfefb3700 1 -- 172.21.15.190:0/3143773581 wait complete. 2024-01-16T17:17:03.763 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:04.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:04 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:03.782851+0000 mon.smithi154 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.190:0/3143773581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:05.118 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:05.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:05.707 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:03.841827+0000 mgr.smithi154.sllivz (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:06.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 -- 172.21.15.190:0/536633030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 msgr2=0x7f70bc0f90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 --2- 172.21.15.190:0/536633030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 0x7f70bc0f90d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f70ac009fb0 tx=0x7f70ac009b30).stop 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 -- 172.21.15.190:0/536633030 shutdown_connections 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 --2- 172.21.15.190:0/536633030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 0x7f70bc0f90d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 -- 172.21.15.190:0/536633030 >> 172.21.15.190:0/536633030 conn(0x7f70bc0f0b70 msgr2=0x7f70bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 -- 172.21.15.190:0/536633030 shutdown_connections 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 -- 172.21.15.190:0/536633030 wait complete. 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 Processor -- start 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 -- start start 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 0x7f70bc0f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:06.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.235+0000 7f70c3e6c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc0f4fd0 con 0x7f70bc060280 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70c1c08700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 0x7f70bc0f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70c1c08700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 0x7f70bc0f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60510/0 (socket says 172.21.15.190:60510) 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70c1c08700 1 -- 172.21.15.190:0/2367206855 learned_addr learned my addr 172.21.15.190:0/2367206855 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70c1c08700 1 -- 172.21.15.190:0/2367206855 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac005040 con 0x7f70bc060280 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70c1c08700 1 --2- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 0x7f70bc0f6b50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f70ac013510 tx=0x7f70ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70b27fc700 1 -- 172.21.15.190:0/2367206855 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70ac009320 con 0x7f70bc060280 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc0f5170 con 0x7f70bc060280 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70b27fc700 1 -- 172.21.15.190:0/2367206855 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70ac013f00 con 0x7f70bc060280 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70b27fc700 1 -- 172.21.15.190:0/2367206855 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70ac014dc0 con 0x7f70bc060280 2024-01-16T17:17:06.218 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.236+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70bc0f5600 con 0x7f70bc060280 2024-01-16T17:17:06.219 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.237+0000 7f70b27fc700 1 -- 172.21.15.190:0/2367206855 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70ac0095f0 con 0x7f70bc060280 2024-01-16T17:17:06.219 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.237+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70bc059100 con 0x7f70bc060280 2024-01-16T17:17:06.219 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.237+0000 7f70b27fc700 1 --2- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f70a802ff60 0x7f70a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:06.219 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.237+0000 7f70b27fc700 1 -- 172.21.15.190:0/2367206855 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70ac041020 con 0x7f70bc060280 2024-01-16T17:17:06.219 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.237+0000 7f70c1407700 1 --2- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f70a802ff60 0x7f70a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:06.219 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.238+0000 7f70c1407700 1 --2- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f70a802ff60 0x7f70a8032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f70b80096a0 tx=0x7f70b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:06.221 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.242+0000 7f70b27fc700 1 -- 172.21.15.190:0/2367206855 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70ac014420 con 0x7f70bc060280 2024-01-16T17:17:06.429 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.451+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70bc045ad0 con 0x7f70bc060280 2024-01-16T17:17:06.430 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.451+0000 7f70b27fc700 1 -- 172.21.15.190:0/2367206855 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f70ac01c920 con 0x7f70bc060280 2024-01-16T17:17:06.430 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:06.431 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:06.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f70a802ff60 msgr2=0x7f70a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:06.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 --2- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f70a802ff60 0x7f70a8032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f70b80096a0 tx=0x7f70b8006b40).stop 2024-01-16T17:17:06.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 msgr2=0x7f70bc0f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:06.432 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 --2- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 0x7f70bc0f6b50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f70ac013510 tx=0x7f70ac00b040).stop 2024-01-16T17:17:06.433 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 shutdown_connections 2024-01-16T17:17:06.433 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 --2- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f70a802ff60 0x7f70a8032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:06.433 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 --2- 172.21.15.190:0/2367206855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70bc060280 0x7f70bc0f6b50 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:06.433 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 >> 172.21.15.190:0/2367206855 conn(0x7f70bc0f0b70 msgr2=0x7f70bc0f1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:06.433 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 shutdown_connections 2024-01-16T17:17:06.433 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:06.453+0000 7f70c3e6c700 1 -- 172.21.15.190:0/2367206855 wait complete. 2024-01-16T17:17:06.433 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:06.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:06.452564+0000 mon.smithi154 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.190:0/2367206855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:07.794 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:07.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:07.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:05.842225+0000 mgr.smithi154.sllivz (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:08.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.922+0000 7f2141c3f700 1 -- 172.21.15.190:0/2557928247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 msgr2=0x7f213c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:08.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.922+0000 7f2141c3f700 1 --2- 172.21.15.190:0/2557928247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 0x7f213c0f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f2130009fb0 tx=0x7f2130009b30).stop 2024-01-16T17:17:08.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.922+0000 7f2141c3f700 1 -- 172.21.15.190:0/2557928247 shutdown_connections 2024-01-16T17:17:08.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.922+0000 7f2141c3f700 1 --2- 172.21.15.190:0/2557928247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 0x7f213c0f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:08.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.922+0000 7f2141c3f700 1 -- 172.21.15.190:0/2557928247 >> 172.21.15.190:0/2557928247 conn(0x7f213c0f0b70 msgr2=0x7f213c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:08.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.922+0000 7f2141c3f700 1 -- 172.21.15.190:0/2557928247 shutdown_connections 2024-01-16T17:17:08.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.922+0000 7f2141c3f700 1 -- 172.21.15.190:0/2557928247 wait complete. 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.923+0000 7f2141c3f700 1 Processor -- start 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.923+0000 7f2141c3f700 1 -- start start 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.923+0000 7f2141c3f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 0x7f213c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.923+0000 7f2141c3f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213c106c00 con 0x7f213c0f5410 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.923+0000 7f213b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 0x7f213c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.923+0000 7f213b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 0x7f213c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60528/0 (socket says 172.21.15.190:60528) 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.923+0000 7f213b7fe700 1 -- 172.21.15.190:0/4063321900 learned_addr learned my addr 172.21.15.190:0/4063321900 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.924+0000 7f213b7fe700 1 -- 172.21.15.190:0/4063321900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130005040 con 0x7f213c0f5410 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.924+0000 7f213b7fe700 1 --2- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 0x7f213c106710 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f2130013510 tx=0x7f213000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.924+0000 7f2123fff700 1 -- 172.21.15.190:0/4063321900 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2130009320 con 0x7f213c0f5410 2024-01-16T17:17:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.924+0000 7f2123fff700 1 -- 172.21.15.190:0/4063321900 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2130013f00 con 0x7f213c0f5410 2024-01-16T17:17:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.924+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f213c106e00 con 0x7f213c0f5410 2024-01-16T17:17:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.924+0000 7f2123fff700 1 -- 172.21.15.190:0/4063321900 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2130014d90 con 0x7f213c0f5410 2024-01-16T17:17:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.924+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213c107290 con 0x7f213c0f5410 2024-01-16T17:17:08.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.925+0000 7f2123fff700 1 -- 172.21.15.190:0/4063321900 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21300263a0 con 0x7f213c0f5410 2024-01-16T17:17:08.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.925+0000 7f2123fff700 1 --2- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f212402ff60 0x7f2124032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:08.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.925+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213c045ad0 con 0x7f213c0f5410 2024-01-16T17:17:08.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.925+0000 7f2123fff700 1 -- 172.21.15.190:0/4063321900 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2130041300 con 0x7f213c0f5410 2024-01-16T17:17:08.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.925+0000 7f213affd700 1 --2- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f212402ff60 0x7f2124032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:08.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.926+0000 7f213affd700 1 --2- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f212402ff60 0x7f2124032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f212c0096a0 tx=0x7f212c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:08.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:08.930+0000 7f2123fff700 1 -- 172.21.15.190:0/4063321900 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2130012330 con 0x7f213c0f5410 2024-01-16T17:17:09.111 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.133+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f213c108140 con 0x7f213c0f5410 2024-01-16T17:17:09.112 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.134+0000 7f2123fff700 1 -- 172.21.15.190:0/4063321900 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f213002b030 con 0x7f213c0f5410 2024-01-16T17:17:09.112 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:09.112 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f212402ff60 msgr2=0x7f2124032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 --2- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f212402ff60 0x7f2124032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f212c0096a0 tx=0x7f212c006b40).stop 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 msgr2=0x7f213c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 --2- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 0x7f213c106710 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f2130013510 tx=0x7f213000b040).stop 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 shutdown_connections 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 --2- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f212402ff60 0x7f2124032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 --2- 172.21.15.190:0/4063321900 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f213c0f5410 0x7f213c106710 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 >> 172.21.15.190:0/4063321900 conn(0x7f213c0f0b70 msgr2=0x7f213c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 shutdown_connections 2024-01-16T17:17:09.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:09.135+0000 7f2141c3f700 1 -- 172.21.15.190:0/4063321900 wait complete. 2024-01-16T17:17:09.115 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:09.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:07.842558+0000 mgr.smithi154.sllivz (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:09.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:09.134732+0000 mon.smithi154 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.190:0/4063321900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:10.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:10.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:11.607 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.627+0000 7f9ce610b700 1 -- 172.21.15.190:0/843366567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 msgr2=0x7f9ce00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:11.607 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.627+0000 7f9ce610b700 1 --2- 172.21.15.190:0/843366567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 0x7f9ce00f4b60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9cd4009fb0 tx=0x7f9cd4009b30).stop 2024-01-16T17:17:11.607 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.627+0000 7f9ce610b700 1 -- 172.21.15.190:0/843366567 shutdown_connections 2024-01-16T17:17:11.607 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.627+0000 7f9ce610b700 1 --2- 172.21.15.190:0/843366567 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 0x7f9ce00f4b60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:11.607 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.627+0000 7f9ce610b700 1 -- 172.21.15.190:0/843366567 >> 172.21.15.190:0/843366567 conn(0x7f9ce00f0260 msgr2=0x7f9ce00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.627+0000 7f9ce610b700 1 -- 172.21.15.190:0/843366567 shutdown_connections 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.627+0000 7f9ce610b700 1 -- 172.21.15.190:0/843366567 wait complete. 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.627+0000 7f9ce610b700 1 Processor -- start 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.628+0000 7f9ce610b700 1 -- start start 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.628+0000 7f9ce610b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 0x7f9ce00f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.628+0000 7f9ce610b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce00f5770 con 0x7f9ce00f4740 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.628+0000 7f9cdf7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 0x7f9ce00f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.628+0000 7f9cdf7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 0x7f9ce00f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60552/0 (socket says 172.21.15.190:60552) 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.628+0000 7f9cdf7fe700 1 -- 172.21.15.190:0/2309282443 learned_addr learned my addr 172.21.15.190:0/2309282443 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.628+0000 7f9cdf7fe700 1 -- 172.21.15.190:0/2309282443 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd4005040 con 0x7f9ce00f4740 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.628+0000 7f9cdf7fe700 1 --2- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 0x7f9ce00f72f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f9cd4013510 tx=0x7f9cd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:11.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.629+0000 7f9cc7fff700 1 -- 172.21.15.190:0/2309282443 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd4009320 con 0x7f9ce00f4740 2024-01-16T17:17:11.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.629+0000 7f9cc7fff700 1 -- 172.21.15.190:0/2309282443 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cd4013f00 con 0x7f9ce00f4740 2024-01-16T17:17:11.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.629+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce00f5910 con 0x7f9ce00f4740 2024-01-16T17:17:11.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.629+0000 7f9cc7fff700 1 -- 172.21.15.190:0/2309282443 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd4014d90 con 0x7f9ce00f4740 2024-01-16T17:17:11.609 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.629+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce00f5da0 con 0x7f9ce00f4740 2024-01-16T17:17:11.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.629+0000 7f9cc7fff700 1 -- 172.21.15.190:0/2309282443 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cd40263a0 con 0x7f9ce00f4740 2024-01-16T17:17:11.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.629+0000 7f9cc7fff700 1 --2- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cc802ffb0 0x7f9cc8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:11.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.630+0000 7f9cc7fff700 1 -- 172.21.15.190:0/2309282443 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cd4040fe0 con 0x7f9ce00f4740 2024-01-16T17:17:11.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.630+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce0059100 con 0x7f9ce00f4740 2024-01-16T17:17:11.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.630+0000 7f9cdeffd700 1 --2- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cc802ffb0 0x7f9cc8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:11.610 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.630+0000 7f9cdeffd700 1 --2- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cc802ffb0 0x7f9cc8032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9cd00096a0 tx=0x7f9cd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:11.613 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.634+0000 7f9cc7fff700 1 -- 172.21.15.190:0/2309282443 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cd4012920 con 0x7f9ce00f4740 2024-01-16T17:17:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.842+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ce0045ad0 con 0x7f9ce00f4740 2024-01-16T17:17:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.842+0000 7f9cc7fff700 1 -- 172.21.15.190:0/2309282443 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9cd402b020 con 0x7f9ce00f4740 2024-01-16T17:17:11.821 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:11.821 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cc802ffb0 msgr2=0x7f9cc8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 --2- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cc802ffb0 0x7f9cc8032470 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f9cd00096a0 tx=0x7f9cd0006b40).stop 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 msgr2=0x7f9ce00f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 --2- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 0x7f9ce00f72f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f9cd4013510 tx=0x7f9cd400b040).stop 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 shutdown_connections 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 --2- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f9cc802ffb0 0x7f9cc8032470 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 --2- 172.21.15.190:0/2309282443 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ce00f4740 0x7f9ce00f72f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 >> 172.21.15.190:0/2309282443 conn(0x7f9ce00f0260 msgr2=0x7f9ce00f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 shutdown_connections 2024-01-16T17:17:11.823 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:11.844+0000 7f9ce610b700 1 -- 172.21.15.190:0/2309282443 wait complete. 2024-01-16T17:17:11.824 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:11.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:09.842899+0000 mgr.smithi154.sllivz (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:12.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:12 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:11.843477+0000 mon.smithi154 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.190:0/2309282443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:13.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:13.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:13.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:11.843295+0000 mgr.smithi154.sllivz (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:14.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.370+0000 7f48c219d700 1 -- 172.21.15.190:0/3190694923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 msgr2=0x7f48bc0f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.370+0000 7f48c219d700 1 --2- 172.21.15.190:0/3190694923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 0x7f48bc0f7d00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f48b0009fb0 tx=0x7f48b0009b30).stop 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.370+0000 7f48c219d700 1 -- 172.21.15.190:0/3190694923 shutdown_connections 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.370+0000 7f48c219d700 1 --2- 172.21.15.190:0/3190694923 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 0x7f48bc0f7d00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.370+0000 7f48c219d700 1 -- 172.21.15.190:0/3190694923 >> 172.21.15.190:0/3190694923 conn(0x7f48bc0f0120 msgr2=0x7f48bc0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.370+0000 7f48c219d700 1 -- 172.21.15.190:0/3190694923 shutdown_connections 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.370+0000 7f48c219d700 1 -- 172.21.15.190:0/3190694923 wait complete. 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.371+0000 7f48c219d700 1 Processor -- start 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.371+0000 7f48c219d700 1 -- start start 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.371+0000 7f48c219d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 0x7f48bc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.371+0000 7f48c219d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc1069d0 con 0x7f48bc0f78e0 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.371+0000 7f48c119b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 0x7f48bc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.371+0000 7f48c119b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 0x7f48bc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:57428/0 (socket says 172.21.15.190:57428) 2024-01-16T17:17:14.352 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.371+0000 7f48c119b700 1 -- 172.21.15.190:0/2260561061 learned_addr learned my addr 172.21.15.190:0/2260561061 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.372+0000 7f48c119b700 1 -- 172.21.15.190:0/2260561061 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0005040 con 0x7f48bc0f78e0 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.372+0000 7f48c119b700 1 --2- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 0x7f48bc1064e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f48b0013510 tx=0x7f48b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.372+0000 7f48adffb700 1 -- 172.21.15.190:0/2260561061 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b0009320 con 0x7f48bc0f78e0 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.372+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc106bd0 con 0x7f48bc0f78e0 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.372+0000 7f48adffb700 1 -- 172.21.15.190:0/2260561061 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48b0013f00 con 0x7f48bc0f78e0 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.372+0000 7f48adffb700 1 -- 172.21.15.190:0/2260561061 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b0014dc0 con 0x7f48bc0f78e0 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.372+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc107060 con 0x7f48bc0f78e0 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.373+0000 7f48adffb700 1 -- 172.21.15.190:0/2260561061 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48b00095f0 con 0x7f48bc0f78e0 2024-01-16T17:17:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.373+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc045ad0 con 0x7f48bc0f78e0 2024-01-16T17:17:14.354 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.373+0000 7f48adffb700 1 --2- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48a802ff60 0x7f48a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:14.354 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.373+0000 7f48adffb700 1 -- 172.21.15.190:0/2260561061 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48b0041100 con 0x7f48bc0f78e0 2024-01-16T17:17:14.354 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.373+0000 7f48c099a700 1 --2- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48a802ff60 0x7f48a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:14.354 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.374+0000 7f48c099a700 1 --2- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48a802ff60 0x7f48a8032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f48b80096a0 tx=0x7f48b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:14.356 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.378+0000 7f48adffb700 1 -- 172.21.15.190:0/2260561061 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48b0014420 con 0x7f48bc0f78e0 2024-01-16T17:17:14.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.583+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48bc107f10 con 0x7f48bc0f78e0 2024-01-16T17:17:14.562 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.583+0000 7f48adffb700 1 -- 172.21.15.190:0/2260561061 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f48b0010230 con 0x7f48bc0f78e0 2024-01-16T17:17:14.562 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:14.562 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48a802ff60 msgr2=0x7f48a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 --2- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48a802ff60 0x7f48a8032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f48b80096a0 tx=0x7f48b8006b40).stop 2024-01-16T17:17:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 msgr2=0x7f48bc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:14.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 --2- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 0x7f48bc1064e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f48b0013510 tx=0x7f48b000b040).stop 2024-01-16T17:17:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 shutdown_connections 2024-01-16T17:17:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 --2- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48a802ff60 0x7f48a8032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 --2- 172.21.15.190:0/2260561061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48bc0f78e0 0x7f48bc1064e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 >> 172.21.15.190:0/2260561061 conn(0x7f48bc0f0120 msgr2=0x7f48bc0f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 shutdown_connections 2024-01-16T17:17:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:14.585+0000 7f48c219d700 1 -- 172.21.15.190:0/2260561061 wait complete. 2024-01-16T17:17:14.565 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:15.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:15.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:15.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:13.843623+0000 mgr.smithi154.sllivz (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:15.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:14.584426+0000 mon.smithi154 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.190:0/2260561061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:16.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.009+0000 7f5313780700 1 -- 172.21.15.190:0/4196493861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 msgr2=0x7f530c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:16.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.009+0000 7f5313780700 1 --2- 172.21.15.190:0/4196493861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 0x7f530c0f4b60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f52fc009fb0 tx=0x7f52fc009b30).stop 2024-01-16T17:17:16.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.009+0000 7f5313780700 1 -- 172.21.15.190:0/4196493861 shutdown_connections 2024-01-16T17:17:16.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.009+0000 7f5313780700 1 --2- 172.21.15.190:0/4196493861 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 0x7f530c0f4b60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:16.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.009+0000 7f5313780700 1 -- 172.21.15.190:0/4196493861 >> 172.21.15.190:0/4196493861 conn(0x7f530c0f0260 msgr2=0x7f530c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:16.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.009+0000 7f5313780700 1 -- 172.21.15.190:0/4196493861 shutdown_connections 2024-01-16T17:17:16.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.009+0000 7f5313780700 1 -- 172.21.15.190:0/4196493861 wait complete. 2024-01-16T17:17:16.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.010+0000 7f5313780700 1 Processor -- start 2024-01-16T17:17:16.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.010+0000 7f5313780700 1 -- start start 2024-01-16T17:17:16.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.010+0000 7f5313780700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 0x7f530c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:16.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.010+0000 7f5313780700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530c106bc0 con 0x7f530c0f4740 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f531151c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 0x7f530c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f531151c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 0x7f530c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:57434/0 (socket says 172.21.15.190:57434) 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f531151c700 1 -- 172.21.15.190:0/1376852122 learned_addr learned my addr 172.21.15.190:0/1376852122 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f531151c700 1 -- 172.21.15.190:0/1376852122 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52fc005040 con 0x7f530c0f4740 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f531151c700 1 --2- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 0x7f530c1066d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f52fc013510 tx=0x7f52fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f5301ffb700 1 -- 172.21.15.190:0/1376852122 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52fc009320 con 0x7f530c0f4740 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f5301ffb700 1 -- 172.21.15.190:0/1376852122 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52fc013f00 con 0x7f530c0f4740 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530c106dc0 con 0x7f530c0f4740 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f5301ffb700 1 -- 172.21.15.190:0/1376852122 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52fc014d90 con 0x7f530c0f4740 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.011+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530c107250 con 0x7f530c0f4740 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.012+0000 7f5301ffb700 1 -- 172.21.15.190:0/1376852122 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52fc014420 con 0x7f530c0f4740 2024-01-16T17:17:16.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.012+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530c045ad0 con 0x7f530c0f4740 2024-01-16T17:17:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.012+0000 7f5301ffb700 1 --2- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f52f802ff60 0x7f52f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.012+0000 7f5301ffb700 1 -- 172.21.15.190:0/1376852122 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52fc0410c0 con 0x7f530c0f4740 2024-01-16T17:17:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.012+0000 7f5310d1b700 1 --2- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f52f802ff60 0x7f52f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:16.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.013+0000 7f5310d1b700 1 --2- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f52f802ff60 0x7f52f8032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f53080096a0 tx=0x7f5308006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:16.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.017+0000 7f5301ffb700 1 -- 172.21.15.190:0/1376852122 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52fc0095f0 con 0x7f530c0f4740 2024-01-16T17:17:17.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.220+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f530c108100 con 0x7f530c0f4740 2024-01-16T17:17:17.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.221+0000 7f5301ffb700 1 -- 172.21.15.190:0/1376852122 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f52fc01db70 con 0x7f530c0f4740 2024-01-16T17:17:17.200 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:17.200 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f52f802ff60 msgr2=0x7f52f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 --2- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f52f802ff60 0x7f52f8032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f53080096a0 tx=0x7f5308006b40).stop 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 msgr2=0x7f530c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 --2- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 0x7f530c1066d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f52fc013510 tx=0x7f52fc00b040).stop 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 shutdown_connections 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 --2- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f52f802ff60 0x7f52f8032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 --2- 172.21.15.190:0/1376852122 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f530c0f4740 0x7f530c1066d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 >> 172.21.15.190:0/1376852122 conn(0x7f530c0f0260 msgr2=0x7f530c0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 shutdown_connections 2024-01-16T17:17:17.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:17.223+0000 7f5313780700 1 -- 172.21.15.190:0/1376852122 wait complete. 2024-01-16T17:17:17.203 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:17.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:15.843912+0000 mgr.smithi154.sllivz (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:17.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:17.222180+0000 mon.smithi154 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.190:0/1376852122' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:18.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:18.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:19.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.708+0000 7ffbe102d700 1 -- 172.21.15.190:0/3291990750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 msgr2=0x7ffbdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:19.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.708+0000 7ffbe102d700 1 --2- 172.21.15.190:0/3291990750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 0x7ffbdc0f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7ffbd0009fb0 tx=0x7ffbd0009b30).stop 2024-01-16T17:17:19.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.708+0000 7ffbe102d700 1 -- 172.21.15.190:0/3291990750 shutdown_connections 2024-01-16T17:17:19.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.708+0000 7ffbe102d700 1 --2- 172.21.15.190:0/3291990750 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 0x7ffbdc0f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:19.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.708+0000 7ffbe102d700 1 -- 172.21.15.190:0/3291990750 >> 172.21.15.190:0/3291990750 conn(0x7ffbdc0f0b70 msgr2=0x7ffbdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:19.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.708+0000 7ffbe102d700 1 -- 172.21.15.190:0/3291990750 shutdown_connections 2024-01-16T17:17:19.687 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.708+0000 7ffbe102d700 1 -- 172.21.15.190:0/3291990750 wait complete. 2024-01-16T17:17:19.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.709+0000 7ffbe102d700 1 Processor -- start 2024-01-16T17:17:19.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.709+0000 7ffbe102d700 1 -- start start 2024-01-16T17:17:19.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.709+0000 7ffbe102d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 0x7ffbdc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:19.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.709+0000 7ffbe102d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc0fb820 con 0x7ffbdc0f5410 2024-01-16T17:17:19.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.709+0000 7ffbdad9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 0x7ffbdc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:19.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.709+0000 7ffbdad9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 0x7ffbdc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:57446/0 (socket says 172.21.15.190:57446) 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.709+0000 7ffbdad9d700 1 -- 172.21.15.190:0/107380873 learned_addr learned my addr 172.21.15.190:0/107380873 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.709+0000 7ffbdad9d700 1 -- 172.21.15.190:0/107380873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd0005040 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.710+0000 7ffbdad9d700 1 --2- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 0x7ffbdc0fd3a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7ffbd0014000 tx=0x7ffbd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.710+0000 7ffbc37fe700 1 -- 172.21.15.190:0/107380873 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbd00095f0 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.710+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbdc0fb9c0 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.710+0000 7ffbc37fe700 1 -- 172.21.15.190:0/107380873 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffbd0007e70 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.710+0000 7ffbc37fe700 1 -- 172.21.15.190:0/107380873 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbd0014ea0 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.710+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbdc0fbe50 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.711+0000 7ffbc37fe700 1 -- 172.21.15.190:0/107380873 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffbd00079c0 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.711+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbdc059100 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.711+0000 7ffbc37fe700 1 --2- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffbc402ff60 0x7ffbc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.711+0000 7ffbc37fe700 1 -- 172.21.15.190:0/107380873 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffbd0040920 con 0x7ffbdc0f5410 2024-01-16T17:17:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.711+0000 7ffbda59c700 1 --2- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffbc402ff60 0x7ffbc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:19.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.711+0000 7ffbda59c700 1 --2- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffbc402ff60 0x7ffbc4032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ffbcc0096a0 tx=0x7ffbcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:19.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.716+0000 7ffbc37fe700 1 -- 172.21.15.190:0/107380873 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffbd000f950 con 0x7ffbdc0f5410 2024-01-16T17:17:19.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.922+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbdc0fae60 con 0x7ffbdc0f5410 2024-01-16T17:17:19.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.923+0000 7ffbc37fe700 1 -- 172.21.15.190:0/107380873 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffbd0010230 con 0x7ffbdc0f5410 2024-01-16T17:17:19.902 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:19.902 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:19.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffbc402ff60 msgr2=0x7ffbc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:19.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 --2- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffbc402ff60 0x7ffbc4032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7ffbcc0096a0 tx=0x7ffbcc006b40).stop 2024-01-16T17:17:19.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 msgr2=0x7ffbdc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:19.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 --2- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 0x7ffbdc0fd3a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7ffbd0014000 tx=0x7ffbd000b040).stop 2024-01-16T17:17:19.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 shutdown_connections 2024-01-16T17:17:19.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 --2- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ffbc402ff60 0x7ffbc4032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:19.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 --2- 172.21.15.190:0/107380873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffbdc0f5410 0x7ffbdc0fd3a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:19.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 >> 172.21.15.190:0/107380873 conn(0x7ffbdc0f0b70 msgr2=0x7ffbdc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:19.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 shutdown_connections 2024-01-16T17:17:19.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:19.925+0000 7ffbe102d700 1 -- 172.21.15.190:0/107380873 wait complete. 2024-01-16T17:17:19.904 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:19.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:17.844301+0000 mgr.smithi154.sllivz (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:20.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:20 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:19.924266+0000 mon.smithi154 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.190:0/107380873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:21.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:21.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:21.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:19.844588+0000 mgr.smithi154.sllivz (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:22.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.441+0000 7fdf69f6f700 1 -- 172.21.15.190:0/2468640749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 msgr2=0x7fdf640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:22.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.441+0000 7fdf69f6f700 1 --2- 172.21.15.190:0/2468640749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 0x7fdf640f90f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fdf58009fb0 tx=0x7fdf58009b30).stop 2024-01-16T17:17:22.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.441+0000 7fdf69f6f700 1 -- 172.21.15.190:0/2468640749 shutdown_connections 2024-01-16T17:17:22.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.441+0000 7fdf69f6f700 1 --2- 172.21.15.190:0/2468640749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 0x7fdf640f90f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:22.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.441+0000 7fdf69f6f700 1 -- 172.21.15.190:0/2468640749 >> 172.21.15.190:0/2468640749 conn(0x7fdf640f0b70 msgr2=0x7fdf640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:22.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.442+0000 7fdf69f6f700 1 -- 172.21.15.190:0/2468640749 shutdown_connections 2024-01-16T17:17:22.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.442+0000 7fdf69f6f700 1 -- 172.21.15.190:0/2468640749 wait complete. 2024-01-16T17:17:22.423 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.442+0000 7fdf69f6f700 1 Processor -- start 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.442+0000 7fdf69f6f700 1 -- start start 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.442+0000 7fdf69f6f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 0x7fdf641066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.442+0000 7fdf69f6f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64106bd0 con 0x7fdf64060280 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf637fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 0x7fdf641066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf637fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 0x7fdf641066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52434/0 (socket says 172.21.15.190:52434) 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf637fe700 1 -- 172.21.15.190:0/4195020486 learned_addr learned my addr 172.21.15.190:0/4195020486 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf637fe700 1 -- 172.21.15.190:0/4195020486 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf58005040 con 0x7fdf64060280 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf637fe700 1 --2- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 0x7fdf641066e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fdf58013510 tx=0x7fdf5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf4bfff700 1 -- 172.21.15.190:0/4195020486 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf58009320 con 0x7fdf64060280 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf64106dd0 con 0x7fdf64060280 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf4bfff700 1 -- 172.21.15.190:0/4195020486 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf58013f00 con 0x7fdf64060280 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf4bfff700 1 -- 172.21.15.190:0/4195020486 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf58014dc0 con 0x7fdf64060280 2024-01-16T17:17:22.424 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.443+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf64107260 con 0x7fdf64060280 2024-01-16T17:17:22.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.444+0000 7fdf4bfff700 1 -- 172.21.15.190:0/4195020486 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf5801d3d0 con 0x7fdf64060280 2024-01-16T17:17:22.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.444+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf64045ad0 con 0x7fdf64060280 2024-01-16T17:17:22.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.444+0000 7fdf4bfff700 1 --2- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdf4c02ff60 0x7fdf4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:22.425 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.444+0000 7fdf4bfff700 1 -- 172.21.15.190:0/4195020486 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf58041380 con 0x7fdf64060280 2024-01-16T17:17:22.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.444+0000 7fdf62ffd700 1 --2- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdf4c02ff60 0x7fdf4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:22.426 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.445+0000 7fdf62ffd700 1 --2- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdf4c02ff60 0x7fdf4c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdf540096a0 tx=0x7fdf54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:22.428 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.449+0000 7fdf4bfff700 1 -- 172.21.15.190:0/4195020486 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf580126f0 con 0x7fdf64060280 2024-01-16T17:17:22.630 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.651+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf64108110 con 0x7fdf64060280 2024-01-16T17:17:22.631 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.651+0000 7fdf4bfff700 1 -- 172.21.15.190:0/4195020486 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf58012330 con 0x7fdf64060280 2024-01-16T17:17:22.631 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:22.632 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdf4c02ff60 msgr2=0x7fdf4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 --2- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdf4c02ff60 0x7fdf4c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fdf540096a0 tx=0x7fdf54006b40).stop 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 msgr2=0x7fdf641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 --2- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 0x7fdf641066e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fdf58013510 tx=0x7fdf5800b040).stop 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 shutdown_connections 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 --2- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fdf4c02ff60 0x7fdf4c032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 --2- 172.21.15.190:0/4195020486 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf64060280 0x7fdf641066e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 >> 172.21.15.190:0/4195020486 conn(0x7fdf640f0b70 msgr2=0x7fdf640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 shutdown_connections 2024-01-16T17:17:22.633 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:22.653+0000 7fdf69f6f700 1 -- 172.21.15.190:0/4195020486 wait complete. 2024-01-16T17:17:22.634 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:23.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:21.844936+0000 mgr.smithi154.sllivz (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:23.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:22.653564+0000 mon.smithi154 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.190:0/4195020486' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:23.988 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:23.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 -- 172.21.15.190:0/1974698067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 msgr2=0x7f57240f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 --2- 172.21.15.190:0/1974698067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 0x7f57240f66c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f5714009fb0 tx=0x7f5714009b30).stop 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 -- 172.21.15.190:0/1974698067 shutdown_connections 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 --2- 172.21.15.190:0/1974698067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 0x7f57240f66c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 -- 172.21.15.190:0/1974698067 >> 172.21.15.190:0/1974698067 conn(0x7f57240f1aa0 msgr2=0x7f57240f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 -- 172.21.15.190:0/1974698067 shutdown_connections 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 -- 172.21.15.190:0/1974698067 wait complete. 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 Processor -- start 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.117+0000 7f5729464700 1 -- start start 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5729464700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 0x7f57241064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5729464700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57241069a0 con 0x7f57240f62a0 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5723fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 0x7f57241064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5723fff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 0x7f57241064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52464/0 (socket says 172.21.15.190:52464) 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5723fff700 1 -- 172.21.15.190:0/1234738253 learned_addr learned my addr 172.21.15.190:0/1234738253 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5723fff700 1 -- 172.21.15.190:0/1234738253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5714005040 con 0x7f57240f62a0 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5723fff700 1 --2- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 0x7f57241064b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f5714013510 tx=0x7f571400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5720ff9700 1 -- 172.21.15.190:0/1234738253 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5714009320 con 0x7f57240f62a0 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5724106ba0 con 0x7f57240f62a0 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5720ff9700 1 -- 172.21.15.190:0/1234738253 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5714013f00 con 0x7f57240f62a0 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5720ff9700 1 -- 172.21.15.190:0/1234738253 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5714014dc0 con 0x7f57240f62a0 2024-01-16T17:17:25.099 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.118+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5724107030 con 0x7f57240f62a0 2024-01-16T17:17:25.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.119+0000 7f5720ff9700 1 -- 172.21.15.190:0/1234738253 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57140095f0 con 0x7f57240f62a0 2024-01-16T17:17:25.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.119+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5724045ad0 con 0x7f57240f62a0 2024-01-16T17:17:25.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.119+0000 7f5720ff9700 1 --2- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:25.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.119+0000 7f5720ff9700 1 -- 172.21.15.190:0/1234738253 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5714041020 con 0x7f57240f62a0 2024-01-16T17:17:25.100 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.120+0000 7f57237fe700 1 --2- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:25.101 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.120+0000 7f57237fe700 1 --2- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f570c02ff60 0x7f570c032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f57180096a0 tx=0x7f5718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:25.103 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.124+0000 7f5720ff9700 1 -- 172.21.15.190:0/1234738253 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5714014420 con 0x7f57240f62a0 2024-01-16T17:17:25.309 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.330+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5724107e80 con 0x7f57240f62a0 2024-01-16T17:17:25.310 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.331+0000 7f5720ff9700 1 -- 172.21.15.190:0/1234738253 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f571401c920 con 0x7f57240f62a0 2024-01-16T17:17:25.310 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:25.310 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f570c02ff60 msgr2=0x7f570c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 --2- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f570c02ff60 0x7f570c032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f57180096a0 tx=0x7f5718006b40).stop 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 msgr2=0x7f57241064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 --2- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 0x7f57241064b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f5714013510 tx=0x7f571400b040).stop 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 shutdown_connections 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 --2- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f570c02ff60 0x7f570c032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 --2- 172.21.15.190:0/1234738253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f57240f62a0 0x7f57241064b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 >> 172.21.15.190:0/1234738253 conn(0x7f57240f1aa0 msgr2=0x7f57240f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 shutdown_connections 2024-01-16T17:17:25.312 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:25.332+0000 7f5729464700 1 -- 172.21.15.190:0/1234738253 wait complete. 2024-01-16T17:17:25.313 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:25.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:23.845331+0000 mgr.smithi154.sllivz (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:25.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:25.332595+0000 mon.smithi154 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.190:0/1234738253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:26.682 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:26.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:27.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.835+0000 7fb131c31700 1 -- 172.21.15.190:0/685586634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 msgr2=0x7fb12c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:27.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.835+0000 7fb131c31700 1 --2- 172.21.15.190:0/685586634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 0x7fb12c0f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fb120009fb0 tx=0x7fb120009b30).stop 2024-01-16T17:17:27.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.835+0000 7fb131c31700 1 -- 172.21.15.190:0/685586634 shutdown_connections 2024-01-16T17:17:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.835+0000 7fb131c31700 1 --2- 172.21.15.190:0/685586634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 0x7fb12c0f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.835+0000 7fb131c31700 1 -- 172.21.15.190:0/685586634 >> 172.21.15.190:0/685586634 conn(0x7fb12c0f0b70 msgr2=0x7fb12c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.835+0000 7fb131c31700 1 -- 172.21.15.190:0/685586634 shutdown_connections 2024-01-16T17:17:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.835+0000 7fb131c31700 1 -- 172.21.15.190:0/685586634 wait complete. 2024-01-16T17:17:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.836+0000 7fb131c31700 1 Processor -- start 2024-01-16T17:17:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.836+0000 7fb131c31700 1 -- start start 2024-01-16T17:17:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.836+0000 7fb131c31700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 0x7fb12c108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.836+0000 7fb131c31700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12c108e00 con 0x7fb12c0f5410 2024-01-16T17:17:27.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.836+0000 7fb12b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 0x7fb12c108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb12b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 0x7fb12c108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52476/0 (socket says 172.21.15.190:52476) 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb12b7fe700 1 -- 172.21.15.190:0/2024102970 learned_addr learned my addr 172.21.15.190:0/2024102970 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb12b7fe700 1 -- 172.21.15.190:0/2024102970 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb120005040 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb12b7fe700 1 --2- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 0x7fb12c108910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb120013510 tx=0x7fb12000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb113fff700 1 -- 172.21.15.190:0/2024102970 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb120009320 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb113fff700 1 -- 172.21.15.190:0/2024102970 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb120013f00 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12c109000 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb113fff700 1 -- 172.21.15.190:0/2024102970 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb120014d90 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.837+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12c109490 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.838+0000 7fb113fff700 1 -- 172.21.15.190:0/2024102970 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1200263a0 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.838+0000 7fb113fff700 1 --2- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb11402ff60 0x7fb114032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.838+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12c045ad0 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.838+0000 7fb113fff700 1 -- 172.21.15.190:0/2024102970 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1200412d0 con 0x7fb12c0f5410 2024-01-16T17:17:27.818 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.838+0000 7fb12affd700 1 --2- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb11402ff60 0x7fb114032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:27.819 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.839+0000 7fb12affd700 1 --2- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb11402ff60 0x7fb114032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb11c0096a0 tx=0x7fb11c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:27.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:27.843+0000 7fb113fff700 1 -- 172.21.15.190:0/2024102970 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb12001ccf0 con 0x7fb12c0f5410 2024-01-16T17:17:27.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:25.845639+0000 mgr.smithi154.sllivz (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:28.030 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.051+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb12c0faf50 con 0x7fb12c0f5410 2024-01-16T17:17:28.031 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb113fff700 1 -- 172.21.15.190:0/2024102970 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb120012330 con 0x7fb12c0f5410 2024-01-16T17:17:28.031 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:28.031 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb11402ff60 msgr2=0x7fb114032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 --2- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb11402ff60 0x7fb114032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb11c0096a0 tx=0x7fb11c006b40).stop 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 msgr2=0x7fb12c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 --2- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 0x7fb12c108910 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fb120013510 tx=0x7fb12000b040).stop 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 shutdown_connections 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 --2- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb11402ff60 0x7fb114032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 --2- 172.21.15.190:0/2024102970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb12c0f5410 0x7fb12c108910 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 >> 172.21.15.190:0/2024102970 conn(0x7fb12c0f0b70 msgr2=0x7fb12c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 shutdown_connections 2024-01-16T17:17:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:28.053+0000 7fb131c31700 1 -- 172.21.15.190:0/2024102970 wait complete. 2024-01-16T17:17:28.034 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:28.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:28 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:28.053762+0000 mon.smithi154 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.190:0/2024102970' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:29.392 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:29.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:29.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:27.845928+0000 mgr.smithi154.sllivz (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.551+0000 7f7ee2211700 1 -- 172.21.15.190:0/3466670855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 msgr2=0x7f7edc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.551+0000 7f7ee2211700 1 --2- 172.21.15.190:0/3466670855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 0x7f7edc0f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f7ed0009fb0 tx=0x7f7ed0009b30).stop 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.551+0000 7f7ee2211700 1 -- 172.21.15.190:0/3466670855 shutdown_connections 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.551+0000 7f7ee2211700 1 --2- 172.21.15.190:0/3466670855 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 0x7f7edc0f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.551+0000 7f7ee2211700 1 -- 172.21.15.190:0/3466670855 >> 172.21.15.190:0/3466670855 conn(0x7f7edc0f1c70 msgr2=0x7f7edc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.551+0000 7f7ee2211700 1 -- 172.21.15.190:0/3466670855 shutdown_connections 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.551+0000 7f7ee2211700 1 -- 172.21.15.190:0/3466670855 wait complete. 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.552+0000 7f7ee2211700 1 Processor -- start 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.552+0000 7f7ee2211700 1 -- start start 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.552+0000 7f7ee2211700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 0x7f7edc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.552+0000 7f7ee2211700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc106c20 con 0x7f7edc0f6510 2024-01-16T17:17:30.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.552+0000 7f7edb7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 0x7f7edc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7edb7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 0x7f7edc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:52488/0 (socket says 172.21.15.190:52488) 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7edb7fe700 1 -- 172.21.15.190:0/101294793 learned_addr learned my addr 172.21.15.190:0/101294793 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7edb7fe700 1 -- 172.21.15.190:0/101294793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed0005040 con 0x7f7edc0f6510 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7edb7fe700 1 --2- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 0x7f7edc106730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f7ed0013510 tx=0x7f7ed000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7ec3fff700 1 -- 172.21.15.190:0/101294793 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ed0009320 con 0x7f7edc0f6510 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7ec3fff700 1 -- 172.21.15.190:0/101294793 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7ed0013f00 con 0x7f7edc0f6510 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc106e20 con 0x7f7edc0f6510 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7ec3fff700 1 -- 172.21.15.190:0/101294793 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ed0014d90 con 0x7f7edc0f6510 2024-01-16T17:17:30.534 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.553+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc1072b0 con 0x7f7edc0f6510 2024-01-16T17:17:30.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.554+0000 7f7ec3fff700 1 -- 172.21.15.190:0/101294793 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ed0014420 con 0x7f7edc0f6510 2024-01-16T17:17:30.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.554+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7edc045ad0 con 0x7f7edc0f6510 2024-01-16T17:17:30.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.554+0000 7f7ec3fff700 1 --2- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7ec402ffb0 0x7f7ec4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:30.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.554+0000 7f7ec3fff700 1 -- 172.21.15.190:0/101294793 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ed0040ff0 con 0x7f7edc0f6510 2024-01-16T17:17:30.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.554+0000 7f7edaffd700 1 --2- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7ec402ffb0 0x7f7ec4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:30.535 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.555+0000 7f7edaffd700 1 --2- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7ec402ffb0 0x7f7ec4032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f7ecc0096a0 tx=0x7f7ecc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:30.538 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.559+0000 7f7ec3fff700 1 -- 172.21.15.190:0/101294793 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ed006d2b0 con 0x7f7edc0f6510 2024-01-16T17:17:30.743 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.763+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7edc0fbf60 con 0x7f7edc0f6510 2024-01-16T17:17:30.743 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.764+0000 7f7ec3fff700 1 -- 172.21.15.190:0/101294793 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7ed00095f0 con 0x7f7edc0f6510 2024-01-16T17:17:30.743 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:30.744 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:30.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7ec402ffb0 msgr2=0x7f7ec4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:30.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 --2- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7ec402ffb0 0x7f7ec4032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f7ecc0096a0 tx=0x7f7ecc006b40).stop 2024-01-16T17:17:30.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 msgr2=0x7f7edc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:30.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 --2- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 0x7f7edc106730 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f7ed0013510 tx=0x7f7ed000b040).stop 2024-01-16T17:17:30.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 shutdown_connections 2024-01-16T17:17:30.745 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 --2- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7ec402ffb0 0x7f7ec4032470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:30.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 --2- 172.21.15.190:0/101294793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7edc0f6510 0x7f7edc106730 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:30.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 >> 172.21.15.190:0/101294793 conn(0x7f7edc0f1c70 msgr2=0x7f7edc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:30.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 shutdown_connections 2024-01-16T17:17:30.746 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:30.766+0000 7f7ee2211700 1 -- 172.21.15.190:0/101294793 wait complete. 2024-01-16T17:17:30.746 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:31.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:29.846266+0000 mgr.smithi154.sllivz (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:31.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:30.766091+0000 mon.smithi154 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.190:0/101294793' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:32.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:32.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.223+0000 7f5f41972700 1 -- 172.21.15.190:0/2588684986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 msgr2=0x7f5f3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.223+0000 7f5f41972700 1 --2- 172.21.15.190:0/2588684986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 0x7f5f3c0f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f5f30009fb0 tx=0x7f5f30009b30).stop 2024-01-16T17:17:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.223+0000 7f5f41972700 1 -- 172.21.15.190:0/2588684986 shutdown_connections 2024-01-16T17:17:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.223+0000 7f5f41972700 1 --2- 172.21.15.190:0/2588684986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 0x7f5f3c0f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.223+0000 7f5f41972700 1 -- 172.21.15.190:0/2588684986 >> 172.21.15.190:0/2588684986 conn(0x7f5f3c0f1c70 msgr2=0x7f5f3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.223+0000 7f5f41972700 1 -- 172.21.15.190:0/2588684986 shutdown_connections 2024-01-16T17:17:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.223+0000 7f5f41972700 1 -- 172.21.15.190:0/2588684986 wait complete. 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.224+0000 7f5f41972700 1 Processor -- start 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.224+0000 7f5f41972700 1 -- start start 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.224+0000 7f5f41972700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 0x7f5f3c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.224+0000 7f5f41972700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c069e20 con 0x7f5f3c0f6510 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f3affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 0x7f5f3c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f3affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 0x7f5f3c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35710/0 (socket says 172.21.15.190:35710) 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f3affd700 1 -- 172.21.15.190:0/1853562934 learned_addr learned my addr 172.21.15.190:0/1853562934 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f3affd700 1 -- 172.21.15.190:0/1853562934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f30005040 con 0x7f5f3c0f6510 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f3affd700 1 --2- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 0x7f5f3c06b9a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5f30013510 tx=0x7f5f3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f237fe700 1 -- 172.21.15.190:0/1853562934 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f30009320 con 0x7f5f3c0f6510 2024-01-16T17:17:33.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3c069fc0 con 0x7f5f3c0f6510 2024-01-16T17:17:33.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f237fe700 1 -- 172.21.15.190:0/1853562934 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f30013f00 con 0x7f5f3c0f6510 2024-01-16T17:17:33.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f237fe700 1 -- 172.21.15.190:0/1853562934 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f30014d90 con 0x7f5f3c0f6510 2024-01-16T17:17:33.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.225+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f3c06a450 con 0x7f5f3c0f6510 2024-01-16T17:17:33.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.226+0000 7f5f237fe700 1 -- 172.21.15.190:0/1853562934 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f300095f0 con 0x7f5f3c0f6510 2024-01-16T17:17:33.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.226+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3c059100 con 0x7f5f3c0f6510 2024-01-16T17:17:33.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.226+0000 7f5f237fe700 1 --2- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f2402ff60 0x7f5f24032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:33.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.226+0000 7f5f237fe700 1 -- 172.21.15.190:0/1853562934 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f30040fc0 con 0x7f5f3c0f6510 2024-01-16T17:17:33.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.227+0000 7f5f3a7fc700 1 --2- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f2402ff60 0x7f5f24032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:33.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.227+0000 7f5f3a7fc700 1 --2- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f2402ff60 0x7f5f24032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5f2c0096a0 tx=0x7f5f2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:33.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.231+0000 7f5f237fe700 1 -- 172.21.15.190:0/1853562934 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f30014420 con 0x7f5f3c0f6510 2024-01-16T17:17:33.413 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.434+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f3c045ad0 con 0x7f5f3c0f6510 2024-01-16T17:17:33.413 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.435+0000 7f5f237fe700 1 -- 172.21.15.190:0/1853562934 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f30010230 con 0x7f5f3c0f6510 2024-01-16T17:17:33.413 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:33.414 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f2402ff60 msgr2=0x7f5f24032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 --2- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f2402ff60 0x7f5f24032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5f2c0096a0 tx=0x7f5f2c006b40).stop 2024-01-16T17:17:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 msgr2=0x7f5f3c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 --2- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 0x7f5f3c06b9a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f5f30013510 tx=0x7f5f3000b040).stop 2024-01-16T17:17:33.416 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 shutdown_connections 2024-01-16T17:17:33.416 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 --2- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f5f2402ff60 0x7f5f24032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:33.416 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 --2- 172.21.15.190:0/1853562934 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f3c0f6510 0x7f5f3c06b9a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:33.416 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 >> 172.21.15.190:0/1853562934 conn(0x7f5f3c0f1c70 msgr2=0x7f5f3c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:33.416 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 shutdown_connections 2024-01-16T17:17:33.416 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:33.437+0000 7f5f41972700 1 -- 172.21.15.190:0/1853562934 wait complete. 2024-01-16T17:17:33.416 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:33.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:31.846566+0000 mgr.smithi154.sllivz (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:33.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:33.436170+0000 mon.smithi154 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.190:0/1853562934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:34.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:34.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:35.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.933+0000 7f4919038700 1 -- 172.21.15.190:0/408112782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 msgr2=0x7f49140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:35.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.933+0000 7f4919038700 1 --2- 172.21.15.190:0/408112782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 0x7f49140f5830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f48fc009fb0 tx=0x7f48fc009b30).stop 2024-01-16T17:17:35.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.933+0000 7f4919038700 1 -- 172.21.15.190:0/408112782 shutdown_connections 2024-01-16T17:17:35.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.933+0000 7f4919038700 1 --2- 172.21.15.190:0/408112782 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 0x7f49140f5830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:35.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.933+0000 7f4919038700 1 -- 172.21.15.190:0/408112782 >> 172.21.15.190:0/408112782 conn(0x7f49140f0b70 msgr2=0x7f49140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:35.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.933+0000 7f4919038700 1 -- 172.21.15.190:0/408112782 shutdown_connections 2024-01-16T17:17:35.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.933+0000 7f4919038700 1 -- 172.21.15.190:0/408112782 wait complete. 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.934+0000 7f4919038700 1 Processor -- start 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.934+0000 7f4919038700 1 -- start start 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.934+0000 7f4919038700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 0x7f4914105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.934+0000 7f4919038700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4914105af0 con 0x7f49140f5410 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f4912d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 0x7f4914105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f4912d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 0x7f4914105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35728/0 (socket says 172.21.15.190:35728) 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f4912d9d700 1 -- 172.21.15.190:0/1339664146 learned_addr learned my addr 172.21.15.190:0/1339664146 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f4912d9d700 1 -- 172.21.15.190:0/1339664146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc005040 con 0x7f49140f5410 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f4912d9d700 1 --2- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 0x7f4914105600 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f48fc013510 tx=0x7f48fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f490b7fe700 1 -- 172.21.15.190:0/1339664146 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48fc009320 con 0x7f49140f5410 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4914105cf0 con 0x7f49140f5410 2024-01-16T17:17:35.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f490b7fe700 1 -- 172.21.15.190:0/1339664146 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48fc013f00 con 0x7f49140f5410 2024-01-16T17:17:35.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f490b7fe700 1 -- 172.21.15.190:0/1339664146 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48fc014d90 con 0x7f49140f5410 2024-01-16T17:17:35.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.935+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4914106180 con 0x7f49140f5410 2024-01-16T17:17:35.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.936+0000 7f490b7fe700 1 -- 172.21.15.190:0/1339664146 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48fc0095f0 con 0x7f49140f5410 2024-01-16T17:17:35.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.936+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4914045ad0 con 0x7f49140f5410 2024-01-16T17:17:35.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.936+0000 7f490b7fe700 1 --2- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f490002ff60 0x7f4900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:35.916 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.936+0000 7f490b7fe700 1 -- 172.21.15.190:0/1339664146 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48fc040fc0 con 0x7f49140f5410 2024-01-16T17:17:35.916 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.937+0000 7f491259c700 1 --2- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f490002ff60 0x7f4900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:35.916 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.937+0000 7f491259c700 1 --2- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f490002ff60 0x7f4900032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f49040096a0 tx=0x7f4904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:35.919 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:35.941+0000 7f490b7fe700 1 -- 172.21.15.190:0/1339664146 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48fc014420 con 0x7f49140f5410 2024-01-16T17:17:35.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:33.846929+0000 mgr.smithi154.sllivz (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:36.127 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.148+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49140fae60 con 0x7f49140f5410 2024-01-16T17:17:36.127 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.149+0000 7f490b7fe700 1 -- 172.21.15.190:0/1339664146 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f48fc010230 con 0x7f49140f5410 2024-01-16T17:17:36.128 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:36.128 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:36.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f490002ff60 msgr2=0x7f4900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:36.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 --2- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f490002ff60 0x7f4900032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f49040096a0 tx=0x7f4904006b40).stop 2024-01-16T17:17:36.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 msgr2=0x7f4914105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:36.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 --2- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 0x7f4914105600 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f48fc013510 tx=0x7f48fc00b040).stop 2024-01-16T17:17:36.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 shutdown_connections 2024-01-16T17:17:36.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 --2- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f490002ff60 0x7f4900032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:36.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 --2- 172.21.15.190:0/1339664146 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49140f5410 0x7f4914105600 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:36.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 >> 172.21.15.190:0/1339664146 conn(0x7f49140f0b70 msgr2=0x7f49140f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:36.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 shutdown_connections 2024-01-16T17:17:36.130 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:36.151+0000 7f4919038700 1 -- 172.21.15.190:0/1339664146 wait complete. 2024-01-16T17:17:36.130 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:36.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:36.150370+0000 mon.smithi154 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.190:0/1339664146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:37.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:37.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:37.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:35.847178+0000 mgr.smithi154.sllivz (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:38.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 -- 172.21.15.190:0/4262652941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 msgr2=0x7fce840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:38.668 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 --2- 172.21.15.190:0/4262652941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 0x7fce840f55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fce80009fb0 tx=0x7fce80009b30).stop 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 -- 172.21.15.190:0/4262652941 shutdown_connections 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 --2- 172.21.15.190:0/4262652941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 0x7fce840f55c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 -- 172.21.15.190:0/4262652941 >> 172.21.15.190:0/4262652941 conn(0x7fce840f09a0 msgr2=0x7fce840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 -- 172.21.15.190:0/4262652941 shutdown_connections 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 -- 172.21.15.190:0/4262652941 wait complete. 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 Processor -- start 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 -- start start 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 0x7fce841064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.689+0000 7fce8a5a7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce841069b0 con 0x7fce840f51a0 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce895a5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 0x7fce841064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce895a5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 0x7fce841064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35746/0 (socket says 172.21.15.190:35746) 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce895a5700 1 -- 172.21.15.190:0/1814770545 learned_addr learned my addr 172.21.15.190:0/1814770545 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce895a5700 1 -- 172.21.15.190:0/1814770545 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce80005040 con 0x7fce840f51a0 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce895a5700 1 --2- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 0x7fce841064c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fce80013510 tx=0x7fce8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:38.669 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce79ffb700 1 -- 172.21.15.190:0/1814770545 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce80009320 con 0x7fce840f51a0 2024-01-16T17:17:38.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce84106bb0 con 0x7fce840f51a0 2024-01-16T17:17:38.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce79ffb700 1 -- 172.21.15.190:0/1814770545 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce80013f00 con 0x7fce840f51a0 2024-01-16T17:17:38.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce79ffb700 1 -- 172.21.15.190:0/1814770545 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce80014dc0 con 0x7fce840f51a0 2024-01-16T17:17:38.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.690+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce84107040 con 0x7fce840f51a0 2024-01-16T17:17:38.670 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.691+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce84045ad0 con 0x7fce840f51a0 2024-01-16T17:17:38.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.691+0000 7fce79ffb700 1 -- 172.21.15.190:0/1814770545 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce800095f0 con 0x7fce840f51a0 2024-01-16T17:17:38.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.691+0000 7fce79ffb700 1 --2- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fce7002ff60 0x7fce70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:38.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.691+0000 7fce79ffb700 1 -- 172.21.15.190:0/1814770545 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce80041020 con 0x7fce840f51a0 2024-01-16T17:17:38.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.691+0000 7fce88da4700 1 --2- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fce7002ff60 0x7fce70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:38.671 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.692+0000 7fce88da4700 1 --2- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fce7002ff60 0x7fce70032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fce740096a0 tx=0x7fce74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.696+0000 7fce79ffb700 1 -- 172.21.15.190:0/1814770545 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce80014420 con 0x7fce840f51a0 2024-01-16T17:17:38.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.906+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce84107e70 con 0x7fce840f51a0 2024-01-16T17:17:38.885 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.907+0000 7fce79ffb700 1 -- 172.21.15.190:0/1814770545 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fce8001c920 con 0x7fce840f51a0 2024-01-16T17:17:38.885 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:38.885 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fce7002ff60 msgr2=0x7fce70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 --2- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fce7002ff60 0x7fce70032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fce740096a0 tx=0x7fce74006b40).stop 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 msgr2=0x7fce841064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 --2- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 0x7fce841064c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fce80013510 tx=0x7fce8000b040).stop 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 shutdown_connections 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 --2- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fce7002ff60 0x7fce70032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 --2- 172.21.15.190:0/1814770545 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fce840f51a0 0x7fce841064c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 >> 172.21.15.190:0/1814770545 conn(0x7fce840f09a0 msgr2=0x7fce840f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 shutdown_connections 2024-01-16T17:17:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:38.908+0000 7fce8a5a7700 1 -- 172.21.15.190:0/1814770545 wait complete. 2024-01-16T17:17:38.888 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:39.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:37.847531+0000 mgr.smithi154.sllivz (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:39.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:38.907791+0000 mon.smithi154 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.190:0/1814770545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:40.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:40.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:41.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.355+0000 7fac532f7700 1 -- 172.21.15.190:0/3614193606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 msgr2=0x7fac4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:41.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.355+0000 7fac532f7700 1 --2- 172.21.15.190:0/3614193606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 0x7fac4c0f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fac3c009fb0 tx=0x7fac3c009b30).stop 2024-01-16T17:17:41.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.355+0000 7fac532f7700 1 -- 172.21.15.190:0/3614193606 shutdown_connections 2024-01-16T17:17:41.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.355+0000 7fac532f7700 1 --2- 172.21.15.190:0/3614193606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 0x7fac4c0f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:41.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.355+0000 7fac532f7700 1 -- 172.21.15.190:0/3614193606 >> 172.21.15.190:0/3614193606 conn(0x7fac4c0f0b70 msgr2=0x7fac4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:41.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.355+0000 7fac532f7700 1 -- 172.21.15.190:0/3614193606 shutdown_connections 2024-01-16T17:17:41.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.355+0000 7fac532f7700 1 -- 172.21.15.190:0/3614193606 wait complete. 2024-01-16T17:17:41.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac532f7700 1 Processor -- start 2024-01-16T17:17:41.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac532f7700 1 -- start start 2024-01-16T17:17:41.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac532f7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 0x7fac4c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:41.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac532f7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c060240 con 0x7fac4c0f5410 2024-01-16T17:17:41.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac51093700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 0x7fac4c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:41.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac51093700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 0x7fac4c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:35756/0 (socket says 172.21.15.190:35756) 2024-01-16T17:17:41.336 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac51093700 1 -- 172.21.15.190:0/3394754519 learned_addr learned my addr 172.21.15.190:0/3394754519 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac51093700 1 -- 172.21.15.190:0/3394754519 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c005040 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.356+0000 7fac51093700 1 --2- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 0x7fac4c109400 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fac3c013510 tx=0x7fac3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.357+0000 7fac41ffb700 1 -- 172.21.15.190:0/3394754519 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac3c009320 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.357+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4c060440 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.357+0000 7fac41ffb700 1 -- 172.21.15.190:0/3394754519 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac3c013f00 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.357+0000 7fac41ffb700 1 -- 172.21.15.190:0/3394754519 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac3c014d90 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.357+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4c109ab0 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.358+0000 7fac41ffb700 1 -- 172.21.15.190:0/3394754519 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac3c0095f0 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.358+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4c045ad0 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.358+0000 7fac41ffb700 1 --2- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac3802ff60 0x7fac38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.358+0000 7fac41ffb700 1 -- 172.21.15.190:0/3394754519 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac3c0410c0 con 0x7fac4c0f5410 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.358+0000 7fac50892700 1 --2- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac3802ff60 0x7fac38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:41.337 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.358+0000 7fac50892700 1 --2- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac3802ff60 0x7fac38032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fac480096a0 tx=0x7fac48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:41.340 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.362+0000 7fac41ffb700 1 -- 172.21.15.190:0/3394754519 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac3c01d7f0 con 0x7fac4c0f5410 2024-01-16T17:17:41.547 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.568+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac4c0fae60 con 0x7fac4c0f5410 2024-01-16T17:17:41.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.569+0000 7fac41ffb700 1 -- 172.21.15.190:0/3394754519 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fac3c010230 con 0x7fac4c0f5410 2024-01-16T17:17:41.548 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:41.548 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:41.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac3802ff60 msgr2=0x7fac38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:41.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 --2- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac3802ff60 0x7fac38032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fac480096a0 tx=0x7fac48006b40).stop 2024-01-16T17:17:41.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 msgr2=0x7fac4c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:41.549 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 --2- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 0x7fac4c109400 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fac3c013510 tx=0x7fac3c00b040).stop 2024-01-16T17:17:41.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 shutdown_connections 2024-01-16T17:17:41.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 --2- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac3802ff60 0x7fac38032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:41.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 --2- 172.21.15.190:0/3394754519 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0f5410 0x7fac4c109400 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:41.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 >> 172.21.15.190:0/3394754519 conn(0x7fac4c0f0b70 msgr2=0x7fac4c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:41.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 shutdown_connections 2024-01-16T17:17:41.550 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:41.571+0000 7fac532f7700 1 -- 172.21.15.190:0/3394754519 wait complete. 2024-01-16T17:17:41.550 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:41.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:39.847899+0000 mgr.smithi154.sllivz (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:42.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:42.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:42.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:42 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:41.570193+0000 mon.smithi154 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.190:0/3394754519' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:43.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:41.848291+0000 mgr.smithi154.sllivz (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:44.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.031+0000 7f306199c700 1 -- 172.21.15.190:0/281414203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 msgr2=0x7f305c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:44.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.031+0000 7f306199c700 1 --2- 172.21.15.190:0/281414203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 0x7f305c0f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f3050009fb0 tx=0x7f3050009b30).stop 2024-01-16T17:17:44.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.031+0000 7f306199c700 1 -- 172.21.15.190:0/281414203 shutdown_connections 2024-01-16T17:17:44.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.031+0000 7f306199c700 1 --2- 172.21.15.190:0/281414203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 0x7f305c0f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:44.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.031+0000 7f306199c700 1 -- 172.21.15.190:0/281414203 >> 172.21.15.190:0/281414203 conn(0x7f305c0f1c70 msgr2=0x7f305c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:44.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.031+0000 7f306199c700 1 -- 172.21.15.190:0/281414203 shutdown_connections 2024-01-16T17:17:44.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.031+0000 7f306199c700 1 -- 172.21.15.190:0/281414203 wait complete. 2024-01-16T17:17:44.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.032+0000 7f306199c700 1 Processor -- start 2024-01-16T17:17:44.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.032+0000 7f306199c700 1 -- start start 2024-01-16T17:17:44.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.032+0000 7f306199c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 0x7f305c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:44.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.032+0000 7f306199c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305c106c20 con 0x7f305c0f6510 2024-01-16T17:17:44.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.032+0000 7f305affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 0x7f305c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:44.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.032+0000 7f305affd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 0x7f305c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:54576/0 (socket says 172.21.15.190:54576) 2024-01-16T17:17:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.032+0000 7f305affd700 1 -- 172.21.15.190:0/1514414699 learned_addr learned my addr 172.21.15.190:0/1514414699 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.033+0000 7f305affd700 1 -- 172.21.15.190:0/1514414699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3050005040 con 0x7f305c0f6510 2024-01-16T17:17:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.033+0000 7f305affd700 1 --2- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 0x7f305c106730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f3050013510 tx=0x7f305000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.033+0000 7f30437fe700 1 -- 172.21.15.190:0/1514414699 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3050009320 con 0x7f305c0f6510 2024-01-16T17:17:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.033+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c106e20 con 0x7f305c0f6510 2024-01-16T17:17:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.033+0000 7f30437fe700 1 -- 172.21.15.190:0/1514414699 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3050013f00 con 0x7f305c0f6510 2024-01-16T17:17:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.033+0000 7f30437fe700 1 -- 172.21.15.190:0/1514414699 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3050014d90 con 0x7f305c0f6510 2024-01-16T17:17:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.033+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305c1072b0 con 0x7f305c0f6510 2024-01-16T17:17:44.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.034+0000 7f30437fe700 1 -- 172.21.15.190:0/1514414699 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3050014420 con 0x7f305c0f6510 2024-01-16T17:17:44.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.034+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305c045ad0 con 0x7f305c0f6510 2024-01-16T17:17:44.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.034+0000 7f30437fe700 1 --2- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f304402ff60 0x7f3044032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:44.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.034+0000 7f30437fe700 1 -- 172.21.15.190:0/1514414699 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30500410d0 con 0x7f305c0f6510 2024-01-16T17:17:44.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.034+0000 7f305a7fc700 1 --2- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f304402ff60 0x7f3044032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:44.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.034+0000 7f305a7fc700 1 --2- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f304402ff60 0x7f3044032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f304c0096a0 tx=0x7f304c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:44.017 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.039+0000 7f30437fe700 1 -- 172.21.15.190:0/1514414699 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30500095f0 con 0x7f305c0f6510 2024-01-16T17:17:44.222 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.243+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f305c0fbf60 con 0x7f305c0f6510 2024-01-16T17:17:44.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.244+0000 7f30437fe700 1 -- 172.21.15.190:0/1514414699 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f305001db70 con 0x7f305c0f6510 2024-01-16T17:17:44.223 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:44.224 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:44.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.245+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f304402ff60 msgr2=0x7f3044032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:44.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 --2- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f304402ff60 0x7f3044032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f304c0096a0 tx=0x7f304c006b40).stop 2024-01-16T17:17:44.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 msgr2=0x7f305c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:44.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 --2- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 0x7f305c106730 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f3050013510 tx=0x7f305000b040).stop 2024-01-16T17:17:44.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 shutdown_connections 2024-01-16T17:17:44.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 --2- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f304402ff60 0x7f3044032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:44.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 --2- 172.21.15.190:0/1514414699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f305c0f6510 0x7f305c106730 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:44.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 >> 172.21.15.190:0/1514414699 conn(0x7f305c0f1c70 msgr2=0x7f305c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:44.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 shutdown_connections 2024-01-16T17:17:44.225 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:44.246+0000 7f306199c700 1 -- 172.21.15.190:0/1514414699 wait complete. 2024-01-16T17:17:44.225 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:44.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:44 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:44.245197+0000 mon.smithi154 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.190:0/1514414699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:45.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:45.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:45.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:43.848660+0000 mgr.smithi154.sllivz (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:46.733 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/3452222473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 msgr2=0x7f1fd00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:46.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 --2- 172.21.15.190:0/3452222473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 0x7f1fd00f66c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f1fc4009fb0 tx=0x7f1fc4009b30).stop 2024-01-16T17:17:46.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/3452222473 shutdown_connections 2024-01-16T17:17:46.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 --2- 172.21.15.190:0/3452222473 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 0x7f1fd00f66c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:46.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/3452222473 >> 172.21.15.190:0/3452222473 conn(0x7f1fd00f1ac0 msgr2=0x7f1fd00f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:46.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/3452222473 shutdown_connections 2024-01-16T17:17:46.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/3452222473 wait complete. 2024-01-16T17:17:46.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 Processor -- start 2024-01-16T17:17:46.734 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.755+0000 7f1fd5f8a700 1 -- start start 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.756+0000 7f1fd5f8a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 0x7f1fd01064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.756+0000 7f1fd5f8a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd0106990 con 0x7f1fd00f62a0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.756+0000 7f1fd4f88700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 0x7f1fd01064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.756+0000 7f1fd4f88700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 0x7f1fd01064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:54600/0 (socket says 172.21.15.190:54600) 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.756+0000 7f1fd4f88700 1 -- 172.21.15.190:0/307189572 learned_addr learned my addr 172.21.15.190:0/307189572 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.756+0000 7f1fd4f88700 1 -- 172.21.15.190:0/307189572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc4005040 con 0x7f1fd00f62a0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.756+0000 7f1fd4f88700 1 --2- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 0x7f1fd01064a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f1fc4009830 tx=0x7f1fc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.756+0000 7f1fcd7fa700 1 -- 172.21.15.190:0/307189572 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc4009320 con 0x7f1fd00f62a0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.757+0000 7f1fcd7fa700 1 -- 172.21.15.190:0/307189572 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fc40136c0 con 0x7f1fd00f62a0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.757+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd0106b90 con 0x7f1fd00f62a0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.757+0000 7f1fcd7fa700 1 -- 172.21.15.190:0/307189572 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc401ee90 con 0x7f1fd00f62a0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.757+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd0107020 con 0x7f1fd00f62a0 2024-01-16T17:17:46.736 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.757+0000 7f1fcd7fa700 1 -- 172.21.15.190:0/307189572 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fc4014440 con 0x7f1fd00f62a0 2024-01-16T17:17:46.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.758+0000 7f1fcd7fa700 1 --2- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1fc002ff10 0x7f1fc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:46.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.758+0000 7f1fcd7fa700 1 -- 172.21.15.190:0/307189572 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fc4040fb0 con 0x7f1fd00f62a0 2024-01-16T17:17:46.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.758+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd0045ad0 con 0x7f1fd00f62a0 2024-01-16T17:17:46.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.758+0000 7f1fcffff700 1 --2- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1fc002ff10 0x7f1fc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:46.737 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.758+0000 7f1fcffff700 1 --2- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1fc002ff10 0x7f1fc00323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1fbc0096a0 tx=0x7f1fbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:46.740 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.762+0000 7f1fcd7fa700 1 -- 172.21.15.190:0/307189572 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fc40273f0 con 0x7f1fd00f62a0 2024-01-16T17:17:46.948 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.970+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fd0107ed0 con 0x7f1fd00f62a0 2024-01-16T17:17:46.949 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.971+0000 7f1fcd7fa700 1 -- 172.21.15.190:0/307189572 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1fc401c320 con 0x7f1fd00f62a0 2024-01-16T17:17:46.949 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:46.949 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.972+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1fc002ff10 msgr2=0x7f1fc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.972+0000 7f1fd5f8a700 1 --2- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1fc002ff10 0x7f1fc00323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f1fbc0096a0 tx=0x7f1fbc006b40).stop 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.972+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 msgr2=0x7f1fd01064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.972+0000 7f1fd5f8a700 1 --2- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 0x7f1fd01064a0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f1fc4009830 tx=0x7f1fc400b040).stop 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.972+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 shutdown_connections 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.972+0000 7f1fd5f8a700 1 --2- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f1fc002ff10 0x7f1fc00323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.972+0000 7f1fd5f8a700 1 --2- 172.21.15.190:0/307189572 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fd00f62a0 0x7f1fd01064a0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.972+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 >> 172.21.15.190:0/307189572 conn(0x7f1fd00f1ac0 msgr2=0x7f1fd00f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.973+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 shutdown_connections 2024-01-16T17:17:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:46.973+0000 7f1fd5f8a700 1 -- 172.21.15.190:0/307189572 wait complete. 2024-01-16T17:17:46.952 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:47.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:45.849000+0000 mgr.smithi154.sllivz (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:47.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:46.971825+0000 mon.smithi154 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.190:0/307189572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:48.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:48.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:49.392 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.414+0000 7f4b8d43b700 1 -- 172.21.15.190:0/2771675514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 msgr2=0x7f4b880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:49.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.414+0000 7f4b8d43b700 1 --2- 172.21.15.190:0/2771675514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 0x7f4b880f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f4b78009fb0 tx=0x7f4b78009b30).stop 2024-01-16T17:17:49.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.414+0000 7f4b8d43b700 1 -- 172.21.15.190:0/2771675514 shutdown_connections 2024-01-16T17:17:49.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.414+0000 7f4b8d43b700 1 --2- 172.21.15.190:0/2771675514 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 0x7f4b880f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:49.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.414+0000 7f4b8d43b700 1 -- 172.21.15.190:0/2771675514 >> 172.21.15.190:0/2771675514 conn(0x7f4b880f0b70 msgr2=0x7f4b880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:49.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.414+0000 7f4b8d43b700 1 -- 172.21.15.190:0/2771675514 shutdown_connections 2024-01-16T17:17:49.393 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.414+0000 7f4b8d43b700 1 -- 172.21.15.190:0/2771675514 wait complete. 2024-01-16T17:17:49.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b8d43b700 1 Processor -- start 2024-01-16T17:17:49.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b8d43b700 1 -- start start 2024-01-16T17:17:49.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b8d43b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 0x7f4b88109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:49.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b8d43b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b88060240 con 0x7f4b880f5410 2024-01-16T17:17:49.394 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b86ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 0x7f4b88109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b86ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 0x7f4b88109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:54606/0 (socket says 172.21.15.190:54606) 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b86ffd700 1 -- 172.21.15.190:0/3144766001 learned_addr learned my addr 172.21.15.190:0/3144766001 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b86ffd700 1 -- 172.21.15.190:0/3144766001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b78005040 con 0x7f4b880f5410 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.415+0000 7f4b86ffd700 1 --2- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 0x7f4b88109400 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4b78014000 tx=0x7f4b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.416+0000 7f4b6f7fe700 1 -- 172.21.15.190:0/3144766001 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b780095f0 con 0x7f4b880f5410 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.416+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b88060440 con 0x7f4b880f5410 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.416+0000 7f4b6f7fe700 1 -- 172.21.15.190:0/3144766001 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b78007e70 con 0x7f4b880f5410 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.416+0000 7f4b6f7fe700 1 -- 172.21.15.190:0/3144766001 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b78014ea0 con 0x7f4b880f5410 2024-01-16T17:17:49.395 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.416+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b88109ab0 con 0x7f4b880f5410 2024-01-16T17:17:49.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.416+0000 7f4b6f7fe700 1 -- 172.21.15.190:0/3144766001 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b78025070 con 0x7f4b880f5410 2024-01-16T17:17:49.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.417+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b88045ad0 con 0x7f4b880f5410 2024-01-16T17:17:49.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.417+0000 7f4b6f7fe700 1 --2- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4b7002ff60 0x7f4b70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:49.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.417+0000 7f4b6f7fe700 1 -- 172.21.15.190:0/3144766001 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b780409e0 con 0x7f4b880f5410 2024-01-16T17:17:49.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.417+0000 7f4b867fc700 1 --2- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4b7002ff60 0x7f4b70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:49.396 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.417+0000 7f4b867fc700 1 --2- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4b7002ff60 0x7f4b70032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f4b7c0096a0 tx=0x7f4b7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:49.400 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.421+0000 7f4b6f7fe700 1 -- 172.21.15.190:0/3144766001 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b7800f4e0 con 0x7f4b880f5410 2024-01-16T17:17:49.605 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.627+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b880fae60 con 0x7f4b880f5410 2024-01-16T17:17:49.606 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.628+0000 7f4b6f7fe700 1 -- 172.21.15.190:0/3144766001 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4b7800f300 con 0x7f4b880f5410 2024-01-16T17:17:49.606 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:49.606 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:49.607 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4b7002ff60 msgr2=0x7f4b70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 --2- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4b7002ff60 0x7f4b70032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f4b7c0096a0 tx=0x7f4b7c006b40).stop 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 msgr2=0x7f4b88109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 --2- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 0x7f4b88109400 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4b78014000 tx=0x7f4b7800b040).stop 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 shutdown_connections 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 --2- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f4b7002ff60 0x7f4b70032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 --2- 172.21.15.190:0/3144766001 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b880f5410 0x7f4b88109400 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 >> 172.21.15.190:0/3144766001 conn(0x7f4b880f0b70 msgr2=0x7f4b880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 shutdown_connections 2024-01-16T17:17:49.608 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:49.629+0000 7f4b8d43b700 1 -- 172.21.15.190:0/3144766001 wait complete. 2024-01-16T17:17:49.609 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:49.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:47.849401+0000 mgr.smithi154.sllivz (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:50.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:50 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:49.628778+0000 mon.smithi154 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.190:0/3144766001' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:50.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:50.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:51.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:49.849717+0000 mgr.smithi154.sllivz (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:52.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.082+0000 7f328eb86700 1 -- 172.21.15.190:0/3775388047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 msgr2=0x7f32880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:52.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.082+0000 7f328eb86700 1 --2- 172.21.15.190:0/3775388047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 0x7f32880f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f327c009fb0 tx=0x7f327c009b30).stop 2024-01-16T17:17:52.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.082+0000 7f328eb86700 1 -- 172.21.15.190:0/3775388047 shutdown_connections 2024-01-16T17:17:52.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.082+0000 7f328eb86700 1 --2- 172.21.15.190:0/3775388047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 0x7f32880f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:52.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.082+0000 7f328eb86700 1 -- 172.21.15.190:0/3775388047 >> 172.21.15.190:0/3775388047 conn(0x7f32880f0b70 msgr2=0x7f32880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:52.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.082+0000 7f328eb86700 1 -- 172.21.15.190:0/3775388047 shutdown_connections 2024-01-16T17:17:52.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.082+0000 7f328eb86700 1 -- 172.21.15.190:0/3775388047 wait complete. 2024-01-16T17:17:52.062 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.083+0000 7f328eb86700 1 Processor -- start 2024-01-16T17:17:52.062 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.083+0000 7f328eb86700 1 -- start start 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.083+0000 7f328eb86700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 0x7f3288105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.083+0000 7f328eb86700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3288105af0 con 0x7f32880f5410 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f328c922700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 0x7f3288105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f328c922700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 0x7f3288105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:54618/0 (socket says 172.21.15.190:54618) 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f328c922700 1 -- 172.21.15.190:0/3081293494 learned_addr learned my addr 172.21.15.190:0/3081293494 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f328c922700 1 -- 172.21.15.190:0/3081293494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327c005040 con 0x7f32880f5410 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f328c922700 1 --2- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 0x7f3288105600 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f327c014000 tx=0x7f327c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f32857fa700 1 -- 172.21.15.190:0/3081293494 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327c0095f0 con 0x7f32880f5410 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3288105cf0 con 0x7f32880f5410 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f32857fa700 1 -- 172.21.15.190:0/3081293494 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f327c007e70 con 0x7f32880f5410 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f32857fa700 1 -- 172.21.15.190:0/3081293494 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f327c014ea0 con 0x7f32880f5410 2024-01-16T17:17:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.084+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3288106180 con 0x7f32880f5410 2024-01-16T17:17:52.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.085+0000 7f32857fa700 1 -- 172.21.15.190:0/3081293494 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f327c027070 con 0x7f32880f5410 2024-01-16T17:17:52.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.085+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3288045ad0 con 0x7f32880f5410 2024-01-16T17:17:52.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.085+0000 7f32857fa700 1 --2- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f327802ff60 0x7f3278032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:52.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.085+0000 7f32857fa700 1 -- 172.21.15.190:0/3081293494 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f327c042780 con 0x7f32880f5410 2024-01-16T17:17:52.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.085+0000 7f3287fff700 1 --2- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f327802ff60 0x7f3278032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:52.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.086+0000 7f3287fff700 1 --2- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f327802ff60 0x7f3278032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f32740096a0 tx=0x7f3274006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:52.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.090+0000 7f32857fa700 1 -- 172.21.15.190:0/3081293494 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f327c027300 con 0x7f32880f5410 2024-01-16T17:17:52.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.295+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32880fae60 con 0x7f32880f5410 2024-01-16T17:17:52.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.295+0000 7f32857fa700 1 -- 172.21.15.190:0/3081293494 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f327c027300 con 0x7f32880f5410 2024-01-16T17:17:52.273 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:52.273 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:52.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f327802ff60 msgr2=0x7f3278032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:52.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 --2- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f327802ff60 0x7f3278032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f32740096a0 tx=0x7f3274006b40).stop 2024-01-16T17:17:52.275 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 msgr2=0x7f3288105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:52.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 --2- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 0x7f3288105600 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f327c014000 tx=0x7f327c00b040).stop 2024-01-16T17:17:52.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 shutdown_connections 2024-01-16T17:17:52.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 --2- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f327802ff60 0x7f3278032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:52.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 --2- 172.21.15.190:0/3081293494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f32880f5410 0x7f3288105600 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:52.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 >> 172.21.15.190:0/3081293494 conn(0x7f32880f0b70 msgr2=0x7f32880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:52.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 shutdown_connections 2024-01-16T17:17:52.276 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:52.297+0000 7f328eb86700 1 -- 172.21.15.190:0/3081293494 wait complete. 2024-01-16T17:17:52.276 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:52.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:52 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:52.296294+0000 mon.smithi154 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.190:0/3081293494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:53.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:53.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:53.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:51.850031+0000 mgr.smithi154.sllivz (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 -- 172.21.15.190:0/4227531995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 msgr2=0x7f7c200f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 --2- 172.21.15.190:0/4227531995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 0x7f7c200f8f20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f7c1c009fb0 tx=0x7f7c1c009b30).stop 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 -- 172.21.15.190:0/4227531995 shutdown_connections 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 --2- 172.21.15.190:0/4227531995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 0x7f7c200f8f20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 -- 172.21.15.190:0/4227531995 >> 172.21.15.190:0/4227531995 conn(0x7f7c200f09a0 msgr2=0x7f7c200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 -- 172.21.15.190:0/4227531995 shutdown_connections 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 -- 172.21.15.190:0/4227531995 wait complete. 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 Processor -- start 2024-01-16T17:17:54.827 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 -- start start 2024-01-16T17:17:54.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 0x7f7c20106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:54.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.848+0000 7f7c26c79700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c201069f0 con 0x7f7c200601e0 2024-01-16T17:17:54.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c25c77700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 0x7f7c20106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:54.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c25c77700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 0x7f7c20106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50582/0 (socket says 172.21.15.190:50582) 2024-01-16T17:17:54.828 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c25c77700 1 -- 172.21.15.190:0/320759565 learned_addr learned my addr 172.21.15.190:0/320759565 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:54.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c25c77700 1 -- 172.21.15.190:0/320759565 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c1c005040 con 0x7f7c200601e0 2024-01-16T17:17:54.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c25c77700 1 --2- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 0x7f7c20106500 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f7c1c013510 tx=0x7f7c1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:54.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c167fc700 1 -- 172.21.15.190:0/320759565 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c009320 con 0x7f7c200601e0 2024-01-16T17:17:54.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c20106bf0 con 0x7f7c200601e0 2024-01-16T17:17:54.829 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c167fc700 1 -- 172.21.15.190:0/320759565 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c1c013f00 con 0x7f7c200601e0 2024-01-16T17:17:54.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c167fc700 1 -- 172.21.15.190:0/320759565 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c1c014d90 con 0x7f7c200601e0 2024-01-16T17:17:54.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.849+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c20107080 con 0x7f7c200601e0 2024-01-16T17:17:54.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.850+0000 7f7c167fc700 1 -- 172.21.15.190:0/320759565 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c1c0095f0 con 0x7f7c200601e0 2024-01-16T17:17:54.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.850+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c20045ad0 con 0x7f7c200601e0 2024-01-16T17:17:54.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.850+0000 7f7c167fc700 1 --2- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7c0c02ff60 0x7f7c0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:54.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.850+0000 7f7c167fc700 1 -- 172.21.15.190:0/320759565 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c1c041080 con 0x7f7c200601e0 2024-01-16T17:17:54.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.850+0000 7f7c25476700 1 --2- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7c0c02ff60 0x7f7c0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:54.830 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.851+0000 7f7c25476700 1 --2- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7c0c02ff60 0x7f7c0c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7c100096a0 tx=0x7f7c10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:54.833 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:54.855+0000 7f7c167fc700 1 -- 172.21.15.190:0/320759565 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c1c014420 con 0x7f7c200601e0 2024-01-16T17:17:55.040 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.062+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c20107f30 con 0x7f7c200601e0 2024-01-16T17:17:55.041 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.063+0000 7f7c167fc700 1 -- 172.21.15.190:0/320759565 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7c1c010230 con 0x7f7c200601e0 2024-01-16T17:17:55.041 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:55.041 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7c0c02ff60 msgr2=0x7f7c0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 --2- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7c0c02ff60 0x7f7c0c032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f7c100096a0 tx=0x7f7c10006b40).stop 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 msgr2=0x7f7c20106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 --2- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 0x7f7c20106500 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f7c1c013510 tx=0x7f7c1c00b040).stop 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 shutdown_connections 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 --2- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7c0c02ff60 0x7f7c0c032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 --2- 172.21.15.190:0/320759565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c200601e0 0x7f7c20106500 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 >> 172.21.15.190:0/320759565 conn(0x7f7c200f09a0 msgr2=0x7f7c200f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 shutdown_connections 2024-01-16T17:17:55.043 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:55.065+0000 7f7c26c79700 1 -- 172.21.15.190:0/320759565 wait complete. 2024-01-16T17:17:55.044 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:55.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:53.850382+0000 mgr.smithi154.sllivz (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:55.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:55.063920+0000 mon.smithi154 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.190:0/320759565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:56.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:56.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:56.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:55.931865+0000 mon.smithi154 (mon.0) 280 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:17:56.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:55.932644+0000 mon.smithi154 (mon.0) 281 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:17:57.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.605+0000 7f45957a7700 1 -- 172.21.15.190:0/1274093010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 msgr2=0x7f45900f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:57.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.605+0000 7f45957a7700 1 --2- 172.21.15.190:0/1274093010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 0x7f45900f6ea0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f4584009fb0 tx=0x7f4584009b30).stop 2024-01-16T17:17:57.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.605+0000 7f45957a7700 1 -- 172.21.15.190:0/1274093010 shutdown_connections 2024-01-16T17:17:57.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.605+0000 7f45957a7700 1 --2- 172.21.15.190:0/1274093010 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 0x7f45900f6ea0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:57.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.605+0000 7f45957a7700 1 -- 172.21.15.190:0/1274093010 >> 172.21.15.190:0/1274093010 conn(0x7f45900f02e0 msgr2=0x7f45900f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:57.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.605+0000 7f45957a7700 1 -- 172.21.15.190:0/1274093010 shutdown_connections 2024-01-16T17:17:57.586 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.605+0000 7f45957a7700 1 -- 172.21.15.190:0/1274093010 wait complete. 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.606+0000 7f45957a7700 1 Processor -- start 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.606+0000 7f45957a7700 1 -- start start 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.606+0000 7f45957a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 0x7f4590106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.606+0000 7f45957a7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4590106bf0 con 0x7f45900f6a80 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.606+0000 7f458effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 0x7f4590106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.606+0000 7f458effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 0x7f4590106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50602/0 (socket says 172.21.15.190:50602) 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.606+0000 7f458effd700 1 -- 172.21.15.190:0/3375595988 learned_addr learned my addr 172.21.15.190:0/3375595988 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.607+0000 7f458effd700 1 -- 172.21.15.190:0/3375595988 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584005040 con 0x7f45900f6a80 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.607+0000 7f458effd700 1 --2- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 0x7f4590106700 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f4584013510 tx=0x7f458400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.607+0000 7f45777fe700 1 -- 172.21.15.190:0/3375595988 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4584009320 con 0x7f45900f6a80 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.607+0000 7f45777fe700 1 -- 172.21.15.190:0/3375595988 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4584013f00 con 0x7f45900f6a80 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.607+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4590106df0 con 0x7f45900f6a80 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.607+0000 7f45777fe700 1 -- 172.21.15.190:0/3375595988 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4584014dc0 con 0x7f45900f6a80 2024-01-16T17:17:57.587 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.607+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4590107280 con 0x7f45900f6a80 2024-01-16T17:17:57.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.608+0000 7f45777fe700 1 -- 172.21.15.190:0/3375595988 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45840095f0 con 0x7f45900f6a80 2024-01-16T17:17:57.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.608+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4590045ad0 con 0x7f45900f6a80 2024-01-16T17:17:57.588 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.608+0000 7f45777fe700 1 --2- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f457802ff60 0x7f4578032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:17:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.608+0000 7f45777fe700 1 -- 172.21.15.190:0/3375595988 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4584041000 con 0x7f45900f6a80 2024-01-16T17:17:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.608+0000 7f458e7fc700 1 --2- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f457802ff60 0x7f4578032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:17:57.589 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.609+0000 7f458e7fc700 1 --2- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f457802ff60 0x7f4578032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f45800096a0 tx=0x7f4580006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:17:57.591 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.612+0000 7f45777fe700 1 -- 172.21.15.190:0/3375595988 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4584014420 con 0x7f45900f6a80 2024-01-16T17:17:57.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.816+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4590108130 con 0x7f45900f6a80 2024-01-16T17:17:57.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.817+0000 7f45777fe700 1 -- 172.21.15.190:0/3375595988 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4584010230 con 0x7f45900f6a80 2024-01-16T17:17:57.796 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:17:57.796 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:17:57.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f457802ff60 msgr2=0x7f4578032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:57.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 --2- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f457802ff60 0x7f4578032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f45800096a0 tx=0x7f4580006b40).stop 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 msgr2=0x7f4590106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 --2- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 0x7f4590106700 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f4584013510 tx=0x7f458400b040).stop 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 shutdown_connections 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 --2- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f457802ff60 0x7f4578032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 --2- 172.21.15.190:0/3375595988 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45900f6a80 0x7f4590106700 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 >> 172.21.15.190:0/3375595988 conn(0x7f45900f02e0 msgr2=0x7f45900f4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 shutdown_connections 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:17:57.818+0000 7f45957a7700 1 -- 172.21.15.190:0/3375595988 wait complete. 2024-01-16T17:17:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:17:57.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:55.850708+0000 mgr.smithi154.sllivz (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:17:58.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:58 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:17:57.818682+0000 mon.smithi154 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.190:0/3375595988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:17:59.145 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:17:59.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:17:59.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:17:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:57.851028+0000 mgr.smithi154.sllivz (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.267+0000 7f0888474700 1 -- 172.21.15.190:0/3492788546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 msgr2=0x7f08800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.267+0000 7f0888474700 1 --2- 172.21.15.190:0/3492788546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 0x7f08800f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f0874009fb0 tx=0x7f0874009b30).stop 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.267+0000 7f0888474700 1 -- 172.21.15.190:0/3492788546 shutdown_connections 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.267+0000 7f0888474700 1 --2- 172.21.15.190:0/3492788546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 0x7f08800f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.267+0000 7f0888474700 1 -- 172.21.15.190:0/3492788546 >> 172.21.15.190:0/3492788546 conn(0x7f08800f1c70 msgr2=0x7f08800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.267+0000 7f0888474700 1 -- 172.21.15.190:0/3492788546 shutdown_connections 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.267+0000 7f0888474700 1 -- 172.21.15.190:0/3492788546 wait complete. 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0888474700 1 Processor -- start 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0888474700 1 -- start start 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0888474700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 0x7f0880106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0888474700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0880106c20 con 0x7f08800f6510 2024-01-16T17:18:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0886210700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 0x7f0880106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:00.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0886210700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 0x7f0880106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:50612/0 (socket says 172.21.15.190:50612) 2024-01-16T17:18:00.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0886210700 1 -- 172.21.15.190:0/2846214280 learned_addr learned my addr 172.21.15.190:0/2846214280 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:00.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0886210700 1 -- 172.21.15.190:0/2846214280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0874005040 con 0x7f08800f6510 2024-01-16T17:18:00.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0886210700 1 --2- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 0x7f0880106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f0874013510 tx=0x7f087400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:00.249 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.268+0000 7f0872ffd700 1 -- 172.21.15.190:0/2846214280 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0874009320 con 0x7f08800f6510 2024-01-16T17:18:00.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.269+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0880106e20 con 0x7f08800f6510 2024-01-16T17:18:00.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.269+0000 7f0872ffd700 1 -- 172.21.15.190:0/2846214280 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0874013f00 con 0x7f08800f6510 2024-01-16T17:18:00.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.269+0000 7f0872ffd700 1 -- 172.21.15.190:0/2846214280 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0874014d90 con 0x7f08800f6510 2024-01-16T17:18:00.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.269+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08801072b0 con 0x7f08800f6510 2024-01-16T17:18:00.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.269+0000 7f0872ffd700 1 -- 172.21.15.190:0/2846214280 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0874014420 con 0x7f08800f6510 2024-01-16T17:18:00.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.269+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0880045ad0 con 0x7f08800f6510 2024-01-16T17:18:00.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.269+0000 7f0872ffd700 1 --2- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f086c02ffb0 0x7f086c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:00.250 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.270+0000 7f0872ffd700 1 -- 172.21.15.190:0/2846214280 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0874041090 con 0x7f08800f6510 2024-01-16T17:18:00.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.270+0000 7f0885a0f700 1 --2- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f086c02ffb0 0x7f086c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:00.251 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.270+0000 7f0885a0f700 1 --2- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f086c02ffb0 0x7f086c032470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f087c0096a0 tx=0x7f087c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:00.253 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.274+0000 7f0872ffd700 1 -- 172.21.15.190:0/2846214280 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f087406d2b0 con 0x7f08800f6510 2024-01-16T17:18:00.458 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.479+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08800fbf60 con 0x7f08800f6510 2024-01-16T17:18:00.459 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.480+0000 7f0872ffd700 1 -- 172.21.15.190:0/2846214280 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f08740095f0 con 0x7f08800f6510 2024-01-16T17:18:00.459 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:00.459 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f086c02ffb0 msgr2=0x7f086c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 --2- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f086c02ffb0 0x7f086c032470 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f087c0096a0 tx=0x7f087c006b40).stop 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 msgr2=0x7f0880106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 --2- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 0x7f0880106730 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f0874013510 tx=0x7f087400b040).stop 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 shutdown_connections 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 --2- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f086c02ffb0 0x7f086c032470 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 --2- 172.21.15.190:0/2846214280 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08800f6510 0x7f0880106730 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 >> 172.21.15.190:0/2846214280 conn(0x7f08800f1c70 msgr2=0x7f08800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 shutdown_connections 2024-01-16T17:18:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:00.481+0000 7f0888474700 1 -- 172.21.15.190:0/2846214280 wait complete. 2024-01-16T17:18:00.462 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:00.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:00 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:00.482040+0000 mon.smithi154 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.190:0/2846214280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:01.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:01.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:01.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:17:59.851405+0000 mgr.smithi154.sllivz (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:02.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.010+0000 7ff27eba0700 1 -- 172.21.15.190:0/173771776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 msgr2=0x7ff2780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:02.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.010+0000 7ff27eba0700 1 --2- 172.21.15.190:0/173771776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 0x7ff2780f55c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7ff268009fb0 tx=0x7ff268009b30).stop 2024-01-16T17:18:02.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.010+0000 7ff27eba0700 1 -- 172.21.15.190:0/173771776 shutdown_connections 2024-01-16T17:18:02.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.010+0000 7ff27eba0700 1 --2- 172.21.15.190:0/173771776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 0x7ff2780f55c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:02.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.010+0000 7ff27eba0700 1 -- 172.21.15.190:0/173771776 >> 172.21.15.190:0/173771776 conn(0x7ff2780f09a0 msgr2=0x7ff2780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:02.991 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.010+0000 7ff27eba0700 1 -- 172.21.15.190:0/173771776 shutdown_connections 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.010+0000 7ff27eba0700 1 -- 172.21.15.190:0/173771776 wait complete. 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.011+0000 7ff27eba0700 1 Processor -- start 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.011+0000 7ff27eba0700 1 -- start start 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.011+0000 7ff27eba0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 0x7ff2781091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.011+0000 7ff27eba0700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff278060220 con 0x7ff2780f51a0 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.011+0000 7ff27db9e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 0x7ff2781091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.011+0000 7ff27db9e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 0x7ff2781091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44882/0 (socket says 172.21.15.190:44882) 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.011+0000 7ff27db9e700 1 -- 172.21.15.190:0/2607922680 learned_addr learned my addr 172.21.15.190:0/2607922680 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.011+0000 7ff27db9e700 1 -- 172.21.15.190:0/2607922680 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff268005040 con 0x7ff2780f51a0 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.012+0000 7ff27db9e700 1 --2- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 0x7ff2781091d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ff268009830 tx=0x7ff26800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.012+0000 7ff26e7fc700 1 -- 172.21.15.190:0/2607922680 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff268009320 con 0x7ff2780f51a0 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.012+0000 7ff26e7fc700 1 -- 172.21.15.190:0/2607922680 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2680136c0 con 0x7ff2780f51a0 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.012+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff278060420 con 0x7ff2780f51a0 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.012+0000 7ff26e7fc700 1 -- 172.21.15.190:0/2607922680 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff26801ee90 con 0x7ff2780f51a0 2024-01-16T17:18:02.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.012+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff278109880 con 0x7ff2780f51a0 2024-01-16T17:18:02.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.013+0000 7ff26e7fc700 1 -- 172.21.15.190:0/2607922680 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff26801e420 con 0x7ff2780f51a0 2024-01-16T17:18:02.993 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.013+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff278059100 con 0x7ff2780f51a0 2024-01-16T17:18:02.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.013+0000 7ff26e7fc700 1 --2- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff26402ff10 0x7ff2640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:02.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.013+0000 7ff26e7fc700 1 -- 172.21.15.190:0/2607922680 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff268040f40 con 0x7ff2780f51a0 2024-01-16T17:18:02.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.013+0000 7ff27d39d700 1 --2- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff26402ff10 0x7ff2640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:02.994 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.014+0000 7ff27d39d700 1 --2- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff26402ff10 0x7ff2640323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff2740096a0 tx=0x7ff274006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:02.997 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.018+0000 7ff26e7fc700 1 -- 172.21.15.190:0/2607922680 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff26806d2b0 con 0x7ff2780f51a0 2024-01-16T17:18:03.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.226+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff278045ad0 con 0x7ff2780f51a0 2024-01-16T17:18:03.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.227+0000 7ff26e7fc700 1 -- 172.21.15.190:0/2607922680 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2680095f0 con 0x7ff2780f51a0 2024-01-16T17:18:03.206 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:03.207 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff26402ff10 msgr2=0x7ff2640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 --2- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff26402ff10 0x7ff2640323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ff2740096a0 tx=0x7ff274006b40).stop 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 msgr2=0x7ff2781091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 --2- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 0x7ff2781091d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7ff268009830 tx=0x7ff26800b040).stop 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 shutdown_connections 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 --2- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7ff26402ff10 0x7ff2640323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 --2- 172.21.15.190:0/2607922680 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff2780f51a0 0x7ff2781091d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 >> 172.21.15.190:0/2607922680 conn(0x7ff2780f09a0 msgr2=0x7ff2780f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 shutdown_connections 2024-01-16T17:18:03.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:03.229+0000 7ff27eba0700 1 -- 172.21.15.190:0/2607922680 wait complete. 2024-01-16T17:18:03.209 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:03.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:01.851817+0000 mgr.smithi154.sllivz (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:03.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:03.229124+0000 mon.smithi154 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.190:0/2607922680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:04.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:04.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.778+0000 7f589be91700 1 -- 172.21.15.190:0/1579224441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 msgr2=0x7f58940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.778+0000 7f589be91700 1 --2- 172.21.15.190:0/1579224441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 0x7f58940f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f5884009fb0 tx=0x7f5884009b30).stop 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.778+0000 7f589be91700 1 -- 172.21.15.190:0/1579224441 shutdown_connections 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.778+0000 7f589be91700 1 --2- 172.21.15.190:0/1579224441 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 0x7f58940f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.778+0000 7f589be91700 1 -- 172.21.15.190:0/1579224441 >> 172.21.15.190:0/1579224441 conn(0x7f58940f0b70 msgr2=0x7f58940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.778+0000 7f589be91700 1 -- 172.21.15.190:0/1579224441 shutdown_connections 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.778+0000 7f589be91700 1 -- 172.21.15.190:0/1579224441 wait complete. 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.779+0000 7f589be91700 1 Processor -- start 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.779+0000 7f589be91700 1 -- start start 2024-01-16T17:18:05.759 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.779+0000 7f589be91700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 0x7f5894106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:05.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.779+0000 7f589be91700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5894106c20 con 0x7f58940f5410 2024-01-16T17:18:05.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.779+0000 7f5899c2d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 0x7f5894106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:05.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.779+0000 7f5899c2d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 0x7f5894106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44904/0 (socket says 172.21.15.190:44904) 2024-01-16T17:18:05.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.779+0000 7f5899c2d700 1 -- 172.21.15.190:0/2777255294 learned_addr learned my addr 172.21.15.190:0/2777255294 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:05.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.780+0000 7f5899c2d700 1 -- 172.21.15.190:0/2777255294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5884005040 con 0x7f58940f5410 2024-01-16T17:18:05.760 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.780+0000 7f5899c2d700 1 --2- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 0x7f5894106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f5884014000 tx=0x7f588400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:05.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.780+0000 7f588a7fc700 1 -- 172.21.15.190:0/2777255294 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58840095f0 con 0x7f58940f5410 2024-01-16T17:18:05.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.780+0000 7f588a7fc700 1 -- 172.21.15.190:0/2777255294 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5884007e70 con 0x7f58940f5410 2024-01-16T17:18:05.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.780+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5894106e20 con 0x7f58940f5410 2024-01-16T17:18:05.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.780+0000 7f588a7fc700 1 -- 172.21.15.190:0/2777255294 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5884014ea0 con 0x7f58940f5410 2024-01-16T17:18:05.761 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.780+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58941072b0 con 0x7f58940f5410 2024-01-16T17:18:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.781+0000 7f588a7fc700 1 -- 172.21.15.190:0/2777255294 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5884025070 con 0x7f58940f5410 2024-01-16T17:18:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.781+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5894045ad0 con 0x7f58940f5410 2024-01-16T17:18:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.781+0000 7f588a7fc700 1 --2- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f588002ffb0 0x7f5880032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.781+0000 7f588a7fc700 1 -- 172.21.15.190:0/2777255294 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f58840409a0 con 0x7f58940f5410 2024-01-16T17:18:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.781+0000 7f589942c700 1 --2- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f588002ffb0 0x7f5880032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:05.762 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.782+0000 7f589942c700 1 --2- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f588002ffb0 0x7f5880032470 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f58900096a0 tx=0x7f5890006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:05.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.787+0000 7f588a7fc700 1 -- 172.21.15.190:0/2777255294 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f588400f4e0 con 0x7f58940f5410 2024-01-16T17:18:05.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:03.852126+0000 mgr.smithi154.sllivz (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:05.977 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:05.999+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58940fae60 con 0x7f58940f5410 2024-01-16T17:18:05.978 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.000+0000 7f588a7fc700 1 -- 172.21.15.190:0/2777255294 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f588400f300 con 0x7f58940f5410 2024-01-16T17:18:05.978 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:05.978 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:05.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f588002ffb0 msgr2=0x7f5880032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:05.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 --2- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f588002ffb0 0x7f5880032470 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f58900096a0 tx=0x7f5890006b40).stop 2024-01-16T17:18:05.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 msgr2=0x7f5894106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:05.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 --2- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 0x7f5894106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f5884014000 tx=0x7f588400b040).stop 2024-01-16T17:18:05.980 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 shutdown_connections 2024-01-16T17:18:05.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 --2- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f588002ffb0 0x7f5880032470 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:05.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 --2- 172.21.15.190:0/2777255294 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58940f5410 0x7f5894106730 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:05.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 >> 172.21.15.190:0/2777255294 conn(0x7f58940f0b70 msgr2=0x7f58940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:05.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 shutdown_connections 2024-01-16T17:18:05.981 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:06.002+0000 7f589be91700 1 -- 172.21.15.190:0/2777255294 wait complete. 2024-01-16T17:18:05.981 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:06.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:06.000912+0000 mon.smithi154 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.190:0/2777255294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:07.356 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:07.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:07.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:05.852462+0000 mgr.smithi154.sllivz (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:08.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.538+0000 7feadb59e700 1 -- 172.21.15.190:0/1152948587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 msgr2=0x7feadc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:08.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.538+0000 7feadb59e700 1 --2- 172.21.15.190:0/1152948587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 0x7feadc0f66c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fead0009fb0 tx=0x7fead0009b30).stop 2024-01-16T17:18:08.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.538+0000 7feadb59e700 1 -- 172.21.15.190:0/1152948587 shutdown_connections 2024-01-16T17:18:08.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.538+0000 7feadb59e700 1 --2- 172.21.15.190:0/1152948587 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 0x7feadc0f66c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:08.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.538+0000 7feadb59e700 1 -- 172.21.15.190:0/1152948587 >> 172.21.15.190:0/1152948587 conn(0x7feadc0f1aa0 msgr2=0x7feadc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:08.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.538+0000 7feadb59e700 1 -- 172.21.15.190:0/1152948587 shutdown_connections 2024-01-16T17:18:08.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.538+0000 7feadb59e700 1 -- 172.21.15.190:0/1152948587 wait complete. 2024-01-16T17:18:08.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.539+0000 7feadb59e700 1 Processor -- start 2024-01-16T17:18:08.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.539+0000 7feadb59e700 1 -- start start 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.539+0000 7feadb59e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 0x7feadc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.539+0000 7feadb59e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feadc1069a0 con 0x7feadc0f62a0 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feada59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 0x7feadc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feada59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 0x7feadc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44924/0 (socket says 172.21.15.190:44924) 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feada59c700 1 -- 172.21.15.190:0/2192175715 learned_addr learned my addr 172.21.15.190:0/2192175715 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feada59c700 1 -- 172.21.15.190:0/2192175715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead0005040 con 0x7feadc0f62a0 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feada59c700 1 --2- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 0x7feadc1064b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fead0013510 tx=0x7fead000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feacaffd700 1 -- 172.21.15.190:0/2192175715 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead0009320 con 0x7feadc0f62a0 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feadc106ba0 con 0x7feadc0f62a0 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feacaffd700 1 -- 172.21.15.190:0/2192175715 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fead0013f00 con 0x7feadc0f62a0 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feacaffd700 1 -- 172.21.15.190:0/2192175715 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fead0014dc0 con 0x7feadc0f62a0 2024-01-16T17:18:08.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.540+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feadc107030 con 0x7feadc0f62a0 2024-01-16T17:18:08.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.541+0000 7feacaffd700 1 -- 172.21.15.190:0/2192175715 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fead00095f0 con 0x7feadc0f62a0 2024-01-16T17:18:08.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.541+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feadc045ad0 con 0x7feadc0f62a0 2024-01-16T17:18:08.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.541+0000 7feacaffd700 1 --2- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7feac402ff60 0x7feac4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:08.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.541+0000 7feacaffd700 1 -- 172.21.15.190:0/2192175715 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fead0041040 con 0x7feadc0f62a0 2024-01-16T17:18:08.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.542+0000 7fead9d9b700 1 --2- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7feac402ff60 0x7feac4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:08.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.542+0000 7fead9d9b700 1 --2- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7feac402ff60 0x7feac4032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7feacc0096a0 tx=0x7feacc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:08.524 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.546+0000 7feacaffd700 1 -- 172.21.15.190:0/2192175715 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fead0014420 con 0x7feadc0f62a0 2024-01-16T17:18:08.732 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.754+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feadc107ee0 con 0x7feadc0f62a0 2024-01-16T17:18:08.733 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.755+0000 7feacaffd700 1 -- 172.21.15.190:0/2192175715 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fead0010230 con 0x7feadc0f62a0 2024-01-16T17:18:08.733 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:08.733 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7feac402ff60 msgr2=0x7feac4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 --2- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7feac402ff60 0x7feac4032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7feacc0096a0 tx=0x7feacc006b40).stop 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 msgr2=0x7feadc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 --2- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 0x7feadc1064b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fead0013510 tx=0x7fead000b040).stop 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 shutdown_connections 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 --2- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7feac402ff60 0x7feac4032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 --2- 172.21.15.190:0/2192175715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feadc0f62a0 0x7feadc1064b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 >> 172.21.15.190:0/2192175715 conn(0x7feadc0f1aa0 msgr2=0x7feadc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 shutdown_connections 2024-01-16T17:18:08.735 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:08.757+0000 7feadb59e700 1 -- 172.21.15.190:0/2192175715 wait complete. 2024-01-16T17:18:08.736 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:09.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:07.852747+0000 mgr.smithi154.sllivz (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:09.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:08.756118+0000 mon.smithi154 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.190:0/2192175715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:10.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:10.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:11.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 -- 172.21.15.190:0/3828511805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 msgr2=0x7f48e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 --2- 172.21.15.190:0/3828511805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 0x7f48e80f55c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f48d8009fb0 tx=0x7f48d8009b30).stop 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 -- 172.21.15.190:0/3828511805 shutdown_connections 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 --2- 172.21.15.190:0/3828511805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 0x7f48e80f55c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 -- 172.21.15.190:0/3828511805 >> 172.21.15.190:0/3828511805 conn(0x7f48e80f09a0 msgr2=0x7f48e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 -- 172.21.15.190:0/3828511805 shutdown_connections 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 -- 172.21.15.190:0/3828511805 wait complete. 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 Processor -- start 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.217+0000 7f48ece6d700 1 -- start start 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48ece6d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 0x7f48e81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:11.196 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48ece6d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e81058a0 con 0x7f48e80f51a0 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48e77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 0x7f48e81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48e77fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 0x7f48e81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:44944/0 (socket says 172.21.15.190:44944) 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48e77fe700 1 -- 172.21.15.190:0/1310895736 learned_addr learned my addr 172.21.15.190:0/1310895736 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48e77fe700 1 -- 172.21.15.190:0/1310895736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d8005040 con 0x7f48e80f51a0 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48e77fe700 1 --2- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 0x7f48e81053b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f48d8013510 tx=0x7f48d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48cffff700 1 -- 172.21.15.190:0/1310895736 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48d8009320 con 0x7f48e80f51a0 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.218+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e8105aa0 con 0x7f48e80f51a0 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.219+0000 7f48cffff700 1 -- 172.21.15.190:0/1310895736 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f48d8013f00 con 0x7f48e80f51a0 2024-01-16T17:18:11.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.219+0000 7f48cffff700 1 -- 172.21.15.190:0/1310895736 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48d8014dc0 con 0x7f48e80f51a0 2024-01-16T17:18:11.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.219+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e8105f30 con 0x7f48e80f51a0 2024-01-16T17:18:11.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.219+0000 7f48cffff700 1 -- 172.21.15.190:0/1310895736 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48d80095f0 con 0x7f48e80f51a0 2024-01-16T17:18:11.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.219+0000 7f48cffff700 1 --2- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48d002ff60 0x7f48d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:11.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.220+0000 7f48cffff700 1 -- 172.21.15.190:0/1310895736 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48d8040710 con 0x7f48e80f51a0 2024-01-16T17:18:11.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.220+0000 7f48e6ffd700 1 --2- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48d002ff60 0x7f48d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:11.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.220+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e8059100 con 0x7f48e80f51a0 2024-01-16T17:18:11.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.220+0000 7f48e6ffd700 1 --2- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48d002ff60 0x7f48d0032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f48dc0096a0 tx=0x7f48dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:11.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.225+0000 7f48cffff700 1 -- 172.21.15.190:0/1310895736 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48d8014420 con 0x7f48e80f51a0 2024-01-16T17:18:11.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.431+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48e8045ad0 con 0x7f48e80f51a0 2024-01-16T17:18:11.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.432+0000 7f48cffff700 1 -- 172.21.15.190:0/1310895736 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f48d802b090 con 0x7f48e80f51a0 2024-01-16T17:18:11.410 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:11.410 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:11.411 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48d002ff60 msgr2=0x7f48d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:11.411 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 --2- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48d002ff60 0x7f48d0032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f48dc0096a0 tx=0x7f48dc006b40).stop 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 msgr2=0x7f48e81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 --2- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 0x7f48e81053b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f48d8013510 tx=0x7f48d800b040).stop 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 shutdown_connections 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 --2- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f48d002ff60 0x7f48d0032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 --2- 172.21.15.190:0/1310895736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48e80f51a0 0x7f48e81053b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 >> 172.21.15.190:0/1310895736 conn(0x7f48e80f09a0 msgr2=0x7f48e80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 shutdown_connections 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:11.433+0000 7f48ece6d700 1 -- 172.21.15.190:0/1310895736 wait complete. 2024-01-16T17:18:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:11.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:09.853050+0000 mgr.smithi154.sllivz (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:11.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:11.432680+0000 mon.smithi154 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.190:0/1310895736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:12.781 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:12.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:13.912 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.934+0000 7fbce7514700 1 -- 172.21.15.190:0/1165890017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 msgr2=0x7fbce00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:13.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.934+0000 7fbce7514700 1 --2- 172.21.15.190:0/1165890017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 0x7fbce00f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fbcd0009fb0 tx=0x7fbcd0009b30).stop 2024-01-16T17:18:13.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.934+0000 7fbce7514700 1 -- 172.21.15.190:0/1165890017 shutdown_connections 2024-01-16T17:18:13.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.934+0000 7fbce7514700 1 --2- 172.21.15.190:0/1165890017 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 0x7fbce00f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:13.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.934+0000 7fbce7514700 1 -- 172.21.15.190:0/1165890017 >> 172.21.15.190:0/1165890017 conn(0x7fbce00f1c70 msgr2=0x7fbce00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:13.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.934+0000 7fbce7514700 1 -- 172.21.15.190:0/1165890017 shutdown_connections 2024-01-16T17:18:13.913 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.934+0000 7fbce7514700 1 -- 172.21.15.190:0/1165890017 wait complete. 2024-01-16T17:18:13.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.935+0000 7fbce7514700 1 Processor -- start 2024-01-16T17:18:13.914 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.935+0000 7fbce7514700 1 -- start start 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.935+0000 7fbce7514700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 0x7fbce0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.935+0000 7fbce7514700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce0106c20 con 0x7fbce00f6510 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.935+0000 7fbce52b0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 0x7fbce0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.935+0000 7fbce52b0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 0x7fbce0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:40960/0 (socket says 172.21.15.190:40960) 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.935+0000 7fbce52b0700 1 -- 172.21.15.190:0/298091921 learned_addr learned my addr 172.21.15.190:0/298091921 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.936+0000 7fbce52b0700 1 -- 172.21.15.190:0/298091921 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd0005040 con 0x7fbce00f6510 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.936+0000 7fbce52b0700 1 --2- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 0x7fbce0106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fbcd0013510 tx=0x7fbcd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.936+0000 7fbcd5ffb700 1 -- 172.21.15.190:0/298091921 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd0009320 con 0x7fbce00f6510 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.936+0000 7fbcd5ffb700 1 -- 172.21.15.190:0/298091921 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbcd0013f00 con 0x7fbce00f6510 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.936+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce0106e20 con 0x7fbce00f6510 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.936+0000 7fbcd5ffb700 1 -- 172.21.15.190:0/298091921 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcd0014d90 con 0x7fbce00f6510 2024-01-16T17:18:13.915 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.936+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce01072b0 con 0x7fbce00f6510 2024-01-16T17:18:13.916 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.937+0000 7fbcd5ffb700 1 -- 172.21.15.190:0/298091921 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbcd0014420 con 0x7fbce00f6510 2024-01-16T17:18:13.916 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.937+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce0045ad0 con 0x7fbce00f6510 2024-01-16T17:18:13.916 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.937+0000 7fbcd5ffb700 1 --2- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbccc02ff60 0x7fbccc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:13.916 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.937+0000 7fbcd5ffb700 1 -- 172.21.15.190:0/298091921 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbcd0041110 con 0x7fbce00f6510 2024-01-16T17:18:13.916 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.937+0000 7fbce4aaf700 1 --2- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbccc02ff60 0x7fbccc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:13.917 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.938+0000 7fbce4aaf700 1 --2- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbccc02ff60 0x7fbccc032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fbcdc0096a0 tx=0x7fbcdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:13.920 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:13.942+0000 7fbcd5ffb700 1 -- 172.21.15.190:0/298091921 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcd00095f0 con 0x7fbce00f6510 2024-01-16T17:18:13.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:11.853426+0000 mgr.smithi154.sllivz (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:14.125 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.147+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbce00fbf60 con 0x7fbce00f6510 2024-01-16T17:18:14.126 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.148+0000 7fbcd5ffb700 1 -- 172.21.15.190:0/298091921 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbcd001db70 con 0x7fbce00f6510 2024-01-16T17:18:14.126 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:14.126 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:14.128 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbccc02ff60 msgr2=0x7fbccc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:14.128 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 --2- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbccc02ff60 0x7fbccc032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fbcdc0096a0 tx=0x7fbcdc006b40).stop 2024-01-16T17:18:14.128 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 msgr2=0x7fbce0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:14.128 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 --2- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 0x7fbce0106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fbcd0013510 tx=0x7fbcd000b040).stop 2024-01-16T17:18:14.128 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 shutdown_connections 2024-01-16T17:18:14.128 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 --2- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbccc02ff60 0x7fbccc032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:14.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 --2- 172.21.15.190:0/298091921 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00f6510 0x7fbce0106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:14.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 >> 172.21.15.190:0/298091921 conn(0x7fbce00f1c70 msgr2=0x7fbce00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:14.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 shutdown_connections 2024-01-16T17:18:14.129 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:14.150+0000 7fbce7514700 1 -- 172.21.15.190:0/298091921 wait complete. 2024-01-16T17:18:14.129 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:14.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:14.148695+0000 mon.smithi154 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.190:0/298091921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:15.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:15.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:15.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:13.853754+0000 mgr.smithi154.sllivz (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:16.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 -- 172.21.15.190:0/1591390467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 msgr2=0x7f7e7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:16.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 --2- 172.21.15.190:0/1591390467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f7e64009fb0 tx=0x7f7e64009b30).stop 2024-01-16T17:18:16.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 -- 172.21.15.190:0/1591390467 shutdown_connections 2024-01-16T17:18:16.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 --2- 172.21.15.190:0/1591390467 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:16.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 -- 172.21.15.190:0/1591390467 >> 172.21.15.190:0/1591390467 conn(0x7f7e7c0f0b70 msgr2=0x7f7e7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:16.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 -- 172.21.15.190:0/1591390467 shutdown_connections 2024-01-16T17:18:16.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 -- 172.21.15.190:0/1591390467 wait complete. 2024-01-16T17:18:16.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 Processor -- start 2024-01-16T17:18:16.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 -- start start 2024-01-16T17:18:16.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:16.872 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.717+0000 7f7e80be6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c0fb820 con 0x7f7e7c0f5410 2024-01-16T17:18:16.873 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e7a59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e7a59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:40972/0 (socket says 172.21.15.190:40972) 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e7a59c700 1 -- 172.21.15.190:0/4245929254 learned_addr learned my addr 172.21.15.190:0/4245929254 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e7a59c700 1 -- 172.21.15.190:0/4245929254 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64005040 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e7a59c700 1 --2- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f7e64013510 tx=0x7f7e6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e72ffd700 1 -- 172.21.15.190:0/4245929254 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e64009320 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7c0fb9c0 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e72ffd700 1 -- 172.21.15.190:0/4245929254 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e64013f00 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e72ffd700 1 -- 172.21.15.190:0/4245929254 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e64014d90 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.718+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7c0fbe50 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.719+0000 7f7e72ffd700 1 -- 172.21.15.190:0/4245929254 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e640263a0 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.719+0000 7f7e72ffd700 1 --2- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7e6802ff60 0x7f7e68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.719+0000 7f7e72ffd700 1 -- 172.21.15.190:0/4245929254 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e640410f0 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.719+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7c059100 con 0x7f7e7c0f5410 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.720+0000 7f7e79d9b700 1 --2- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7e6802ff60 0x7f7e68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:16.874 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.720+0000 7f7e79d9b700 1 --2- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7e6802ff60 0x7f7e68032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7e6c0096a0 tx=0x7f7e6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:16.875 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.724+0000 7f7e72ffd700 1 -- 172.21.15.190:0/4245929254 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e64012920 con 0x7f7e7c0f5410 2024-01-16T17:18:16.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.928+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e7c0fc550 con 0x7f7e7c0f5410 2024-01-16T17:18:16.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.928+0000 7f7e72ffd700 1 -- 172.21.15.190:0/4245929254 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7e6402b020 con 0x7f7e7c0f5410 2024-01-16T17:18:16.907 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:16.907 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:16.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7e6802ff60 msgr2=0x7f7e68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:16.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 --2- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7e6802ff60 0x7f7e68032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7e6c0096a0 tx=0x7f7e6c006b40).stop 2024-01-16T17:18:16.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 msgr2=0x7f7e7c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:16.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 --2- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f7e64013510 tx=0x7f7e6400b040).stop 2024-01-16T17:18:16.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 shutdown_connections 2024-01-16T17:18:16.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 --2- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f7e6802ff60 0x7f7e68032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:16.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 --2- 172.21.15.190:0/4245929254 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e7c0f5410 0x7f7e7c0fd3a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:16.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 >> 172.21.15.190:0/4245929254 conn(0x7f7e7c0f0b70 msgr2=0x7f7e7c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:16.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 shutdown_connections 2024-01-16T17:18:16.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:16.930+0000 7f7e80be6700 1 -- 172.21.15.190:0/4245929254 wait complete. 2024-01-16T17:18:16.909 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:17.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:15.854095+0000 mgr.smithi154.sllivz (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:17.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:16.929669+0000 mon.smithi154 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.190:0/4245929254' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:18.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:18.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:19.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.460+0000 7fa3d2db1700 1 -- 172.21.15.190:0/2267901702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 msgr2=0x7fa3cc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:19.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.460+0000 7fa3d2db1700 1 --2- 172.21.15.190:0/2267901702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 0x7fa3cc0f4b60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fa3c0009fb0 tx=0x7fa3c0009b30).stop 2024-01-16T17:18:19.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.460+0000 7fa3d2db1700 1 -- 172.21.15.190:0/2267901702 shutdown_connections 2024-01-16T17:18:19.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.460+0000 7fa3d2db1700 1 --2- 172.21.15.190:0/2267901702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 0x7fa3cc0f4b60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:19.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.460+0000 7fa3d2db1700 1 -- 172.21.15.190:0/2267901702 >> 172.21.15.190:0/2267901702 conn(0x7fa3cc0f0260 msgr2=0x7fa3cc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:19.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.460+0000 7fa3d2db1700 1 -- 172.21.15.190:0/2267901702 shutdown_connections 2024-01-16T17:18:19.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.460+0000 7fa3d2db1700 1 -- 172.21.15.190:0/2267901702 wait complete. 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.461+0000 7fa3d2db1700 1 Processor -- start 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.461+0000 7fa3d2db1700 1 -- start start 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.461+0000 7fa3d2db1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 0x7fa3cc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.461+0000 7fa3d2db1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc106bc0 con 0x7fa3cc0f4740 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.461+0000 7fa3d0b4d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 0x7fa3cc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.461+0000 7fa3d0b4d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 0x7fa3cc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:40992/0 (socket says 172.21.15.190:40992) 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.461+0000 7fa3d0b4d700 1 -- 172.21.15.190:0/1482453095 learned_addr learned my addr 172.21.15.190:0/1482453095 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.462+0000 7fa3d0b4d700 1 -- 172.21.15.190:0/1482453095 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c0005040 con 0x7fa3cc0f4740 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.462+0000 7fa3d0b4d700 1 --2- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 0x7fa3cc1066d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fa3c0013510 tx=0x7fa3c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.462+0000 7fa3c97fa700 1 -- 172.21.15.190:0/1482453095 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3c0009320 con 0x7fa3cc0f4740 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.462+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc106dc0 con 0x7fa3cc0f4740 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.462+0000 7fa3c97fa700 1 -- 172.21.15.190:0/1482453095 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3c0013f00 con 0x7fa3cc0f4740 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.462+0000 7fa3c97fa700 1 -- 172.21.15.190:0/1482453095 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3c0014d90 con 0x7fa3cc0f4740 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.462+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc107250 con 0x7fa3cc0f4740 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.463+0000 7fa3c97fa700 1 -- 172.21.15.190:0/1482453095 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3c0014420 con 0x7fa3cc0f4740 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.463+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc045ad0 con 0x7fa3cc0f4740 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.463+0000 7fa3c97fa700 1 --2- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3bc02ff60 0x7fa3bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.463+0000 7fa3c97fa700 1 -- 172.21.15.190:0/1482453095 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3c00410c0 con 0x7fa3cc0f4740 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.463+0000 7fa3cbfff700 1 --2- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3bc02ff60 0x7fa3bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:19.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.464+0000 7fa3cbfff700 1 --2- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3bc02ff60 0x7fa3bc032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fa3b80096a0 tx=0x7fa3b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:19.446 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.468+0000 7fa3c97fa700 1 -- 172.21.15.190:0/1482453095 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3c00095f0 con 0x7fa3cc0f4740 2024-01-16T17:18:19.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.673+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3cc108100 con 0x7fa3cc0f4740 2024-01-16T17:18:19.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.674+0000 7fa3c97fa700 1 -- 172.21.15.190:0/1482453095 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa3c001db70 con 0x7fa3cc0f4740 2024-01-16T17:18:19.653 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:19.653 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3bc02ff60 msgr2=0x7fa3bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 --2- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3bc02ff60 0x7fa3bc032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fa3b80096a0 tx=0x7fa3b8006b40).stop 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 msgr2=0x7fa3cc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 --2- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 0x7fa3cc1066d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fa3c0013510 tx=0x7fa3c000b040).stop 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 shutdown_connections 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 --2- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa3bc02ff60 0x7fa3bc032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 --2- 172.21.15.190:0/1482453095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3cc0f4740 0x7fa3cc1066d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 >> 172.21.15.190:0/1482453095 conn(0x7fa3cc0f0260 msgr2=0x7fa3cc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:19.654 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 shutdown_connections 2024-01-16T17:18:19.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:19.676+0000 7fa3d2db1700 1 -- 172.21.15.190:0/1482453095 wait complete. 2024-01-16T17:18:19.655 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:19.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:17.854393+0000 mgr.smithi154.sllivz (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:20.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:20 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:19.675180+0000 mon.smithi154 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.190:0/1482453095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:21.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:21.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:21.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:19.854720+0000 mgr.smithi154.sllivz (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:22.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 -- 172.21.15.190:0/779033573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 msgr2=0x7fc7c00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:22.139 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 --2- 172.21.15.190:0/779033573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 0x7fc7c00f66c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fc7b0009fb0 tx=0x7fc7b0009b30).stop 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 -- 172.21.15.190:0/779033573 shutdown_connections 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 --2- 172.21.15.190:0/779033573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 0x7fc7c00f66c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 -- 172.21.15.190:0/779033573 >> 172.21.15.190:0/779033573 conn(0x7fc7c00f1aa0 msgr2=0x7fc7c00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 -- 172.21.15.190:0/779033573 shutdown_connections 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 -- 172.21.15.190:0/779033573 wait complete. 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 Processor -- start 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.161+0000 7fc7c53d8700 1 -- start start 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7c53d8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 0x7fc7c01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7c53d8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c01069a0 con 0x7fc7c00f62a0 2024-01-16T17:18:22.141 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7bffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 0x7fc7c01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7bffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 0x7fc7c01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:41008/0 (socket says 172.21.15.190:41008) 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7bffff700 1 -- 172.21.15.190:0/1274173147 learned_addr learned my addr 172.21.15.190:0/1274173147 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7bffff700 1 -- 172.21.15.190:0/1274173147 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b0005040 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7bffff700 1 --2- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 0x7fc7c01064b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fc7b0013510 tx=0x7fc7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7bcff9700 1 -- 172.21.15.190:0/1274173147 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7b0009320 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7bcff9700 1 -- 172.21.15.190:0/1274173147 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7b0013f00 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c0106ba0 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7bcff9700 1 -- 172.21.15.190:0/1274173147 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7b0014dc0 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.162+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c0107030 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.163+0000 7fc7bcff9700 1 -- 172.21.15.190:0/1274173147 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7b001d3d0 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.163+0000 7fc7bcff9700 1 --2- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc7a802ff60 0x7fc7a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.163+0000 7fc7bcff9700 1 -- 172.21.15.190:0/1274173147 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7b0041030 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.163+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c0045ad0 con 0x7fc7c00f62a0 2024-01-16T17:18:22.142 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.164+0000 7fc7bf7fe700 1 --2- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc7a802ff60 0x7fc7a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:22.143 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.164+0000 7fc7bf7fe700 1 --2- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc7a802ff60 0x7fc7a8032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc7b40096a0 tx=0x7fc7b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:22.146 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.168+0000 7fc7bcff9700 1 -- 172.21.15.190:0/1274173147 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7b0012920 con 0x7fc7c00f62a0 2024-01-16T17:18:22.356 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.378+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7c0107d50 con 0x7fc7c00f62a0 2024-01-16T17:18:22.356 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.378+0000 7fc7bcff9700 1 -- 172.21.15.190:0/1274173147 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc7b0012330 con 0x7fc7c00f62a0 2024-01-16T17:18:22.356 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:22.357 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:22.358 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc7a802ff60 msgr2=0x7fc7a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 --2- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc7a802ff60 0x7fc7a8032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fc7b40096a0 tx=0x7fc7b4006b40).stop 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 msgr2=0x7fc7c01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 --2- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 0x7fc7c01064b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fc7b0013510 tx=0x7fc7b000b040).stop 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 shutdown_connections 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 --2- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fc7a802ff60 0x7fc7a8032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 --2- 172.21.15.190:0/1274173147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7c00f62a0 0x7fc7c01064b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 >> 172.21.15.190:0/1274173147 conn(0x7fc7c00f1aa0 msgr2=0x7fc7c00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 shutdown_connections 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:22.380+0000 7fc7c53d8700 1 -- 172.21.15.190:0/1274173147 wait complete. 2024-01-16T17:18:22.359 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:22.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:22 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:22.379493+0000 mon.smithi154 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.190:0/1274173147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:23.747 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:23.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:23.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:21.855022+0000 mgr.smithi154.sllivz (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:24.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.926+0000 7f0e5bac9700 1 -- 172.21.15.190:0/3599743955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 msgr2=0x7f0e540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:24.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.926+0000 7f0e5bac9700 1 --2- 172.21.15.190:0/3599743955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 0x7f0e540f90f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f0e50009fb0 tx=0x7f0e50009b30).stop 2024-01-16T17:18:24.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 -- 172.21.15.190:0/3599743955 shutdown_connections 2024-01-16T17:18:24.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 --2- 172.21.15.190:0/3599743955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 0x7f0e540f90f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:24.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 -- 172.21.15.190:0/3599743955 >> 172.21.15.190:0/3599743955 conn(0x7f0e540f0b70 msgr2=0x7f0e540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:24.906 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 -- 172.21.15.190:0/3599743955 shutdown_connections 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 -- 172.21.15.190:0/3599743955 wait complete. 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 Processor -- start 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 -- start start 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 0x7f0e540f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e5bac9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e540f4ff0 con 0x7f0e54060280 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e59865700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 0x7f0e540f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e59865700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 0x7f0e540f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58046/0 (socket says 172.21.15.190:58046) 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.927+0000 7f0e59865700 1 -- 172.21.15.190:0/818932094 learned_addr learned my addr 172.21.15.190:0/818932094 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.928+0000 7f0e59865700 1 -- 172.21.15.190:0/818932094 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50005040 con 0x7f0e54060280 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.928+0000 7f0e59865700 1 --2- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 0x7f0e540f6b70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f0e50009830 tx=0x7f0e5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.928+0000 7f0e4a7fc700 1 -- 172.21.15.190:0/818932094 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e50009320 con 0x7f0e54060280 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.928+0000 7f0e4a7fc700 1 -- 172.21.15.190:0/818932094 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0e500136c0 con 0x7f0e54060280 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.928+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e540f5190 con 0x7f0e54060280 2024-01-16T17:18:24.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.928+0000 7f0e4a7fc700 1 -- 172.21.15.190:0/818932094 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e5001ee90 con 0x7f0e54060280 2024-01-16T17:18:24.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.928+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e540f5620 con 0x7f0e54060280 2024-01-16T17:18:24.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.929+0000 7f0e4a7fc700 1 -- 172.21.15.190:0/818932094 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e5001e420 con 0x7f0e54060280 2024-01-16T17:18:24.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.929+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e54059100 con 0x7f0e54060280 2024-01-16T17:18:24.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.929+0000 7f0e4a7fc700 1 --2- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e4002ff10 0x7f0e400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:24.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.929+0000 7f0e4a7fc700 1 -- 172.21.15.190:0/818932094 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e50040f30 con 0x7f0e54060280 2024-01-16T17:18:24.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.929+0000 7f0e59064700 1 --2- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e4002ff10 0x7f0e400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:24.909 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.930+0000 7f0e59064700 1 --2- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e4002ff10 0x7f0e400323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f0e440096a0 tx=0x7f0e44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:24.911 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:24.934+0000 7f0e4a7fc700 1 -- 172.21.15.190:0/818932094 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e5006d2b0 con 0x7f0e54060280 2024-01-16T17:18:25.118 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.140+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e540fbf50 con 0x7f0e54060280 2024-01-16T17:18:25.118 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.140+0000 7f0e4a7fc700 1 -- 172.21.15.190:0/818932094 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0e5001c790 con 0x7f0e54060280 2024-01-16T17:18:25.119 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:25.119 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:25.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e4002ff10 msgr2=0x7f0e400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:25.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 --2- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e4002ff10 0x7f0e400323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f0e440096a0 tx=0x7f0e44006b40).stop 2024-01-16T17:18:25.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 msgr2=0x7f0e540f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:25.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 --2- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 0x7f0e540f6b70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f0e50009830 tx=0x7f0e5000b040).stop 2024-01-16T17:18:25.120 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 shutdown_connections 2024-01-16T17:18:25.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 --2- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0e4002ff10 0x7f0e400323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:25.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 --2- 172.21.15.190:0/818932094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e54060280 0x7f0e540f6b70 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:25.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 >> 172.21.15.190:0/818932094 conn(0x7f0e540f0b70 msgr2=0x7f0e540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:25.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 shutdown_connections 2024-01-16T17:18:25.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:25.142+0000 7f0e5bac9700 1 -- 172.21.15.190:0/818932094 wait complete. 2024-01-16T17:18:25.121 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:25.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:23.855382+0000 mgr.smithi154.sllivz (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:25.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:25.141517+0000 mon.smithi154 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.190:0/818932094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:26.487 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:26.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:26.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:18:24.769039+0000 mgr.smithi154.sllivz (mgr.14162) 231 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi154 ... 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Verifying port 9100 ... 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Non-zero exit code 1 from systemctl start ceph-059f8940-b492-11ee-95ae-87774f69a715@node-exporter.smithi154 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: systemctl: stderr Job for ceph-059f8940-b492-11ee-95ae-87774f69a715@node-exporter.smithi154.service failed because a timeout was exceeded. 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: systemctl: stderr See "systemctl status ceph-059f8940-b492-11ee-95ae-87774f69a715@node-exporter.smithi154.service" and "journalctl -xe" for details. 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Traceback (most recent call last): 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: main() 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: r = ctx.func(ctx) 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: return func(ctx) 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-16T17:18:26.957 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ports=daemon_ports) 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: RuntimeError: Failed command: systemctl start ceph-059f8940-b492-11ee-95ae-87774f69a715@node-exporter.smithi154 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Traceback (most recent call last): 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: yield (conn, connr) 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: code, '\n'.join(err))) 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi154 ... 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Verifying port 9100 ... 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Non-zero exit code 1 from systemctl start ceph-059f8940-b492-11ee-95ae-87774f69a715@node-exporter.smithi154 2024-01-16T17:18:26.958 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: systemctl: stderr Job for ceph-059f8940-b492-11ee-95ae-87774f69a715@node-exporter.smithi154.service failed because a timeout was exceeded. 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: systemctl: stderr See "systemctl status ceph-059f8940-b492-11ee-95ae-87774f69a715@node-exporter.smithi154.service" and "journalctl -xe" for details. 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: Traceback (most recent call last): 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: main() 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: r = ctx.func(ctx) 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: return func(ctx) 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: ports=daemon_ports) 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: File "/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-16T17:18:26.959 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-16T17:18:26.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: RuntimeError: Failed command: systemctl start ceph-059f8940-b492-11ee-95ae-87774f69a715@node-exporter.smithi154 2024-01-16T17:18:26.960 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:26 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cephadm 2024-01-16T17:18:24.774772+0000 mgr.smithi154.sllivz (mgr.14162) 232 : cephadm [INF] Deploying daemon prometheus.smithi154 on smithi154 2024-01-16T17:18:27.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 -- 172.21.15.190:0/456011099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 msgr2=0x7f76bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:27.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 --2- 172.21.15.190:0/456011099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 0x7f76bc0f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f76ac009fb0 tx=0x7f76ac009b30).stop 2024-01-16T17:18:27.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 -- 172.21.15.190:0/456011099 shutdown_connections 2024-01-16T17:18:27.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 --2- 172.21.15.190:0/456011099 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 0x7f76bc0f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 -- 172.21.15.190:0/456011099 >> 172.21.15.190:0/456011099 conn(0x7f76bc0f0b70 msgr2=0x7f76bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 -- 172.21.15.190:0/456011099 shutdown_connections 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 -- 172.21.15.190:0/456011099 wait complete. 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 Processor -- start 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 -- start start 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 0x7f76bc106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.657+0000 7f76c3d34700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc106c00 con 0x7f76bc0f5410 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76c1ad0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 0x7f76bc106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76c1ad0700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 0x7f76bc106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58058/0 (socket says 172.21.15.190:58058) 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76c1ad0700 1 -- 172.21.15.190:0/3497823247 learned_addr learned my addr 172.21.15.190:0/3497823247 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76c1ad0700 1 -- 172.21.15.190:0/3497823247 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac005040 con 0x7f76bc0f5410 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76c1ad0700 1 --2- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 0x7f76bc106710 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f76ac013510 tx=0x7f76ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76b27fc700 1 -- 172.21.15.190:0/3497823247 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac009320 con 0x7f76bc0f5410 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc106e00 con 0x7f76bc0f5410 2024-01-16T17:18:27.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76b27fc700 1 -- 172.21.15.190:0/3497823247 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76ac013f00 con 0x7f76bc0f5410 2024-01-16T17:18:27.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76b27fc700 1 -- 172.21.15.190:0/3497823247 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76ac014d90 con 0x7f76bc0f5410 2024-01-16T17:18:27.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.658+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc107290 con 0x7f76bc0f5410 2024-01-16T17:18:27.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.659+0000 7f76b27fc700 1 -- 172.21.15.190:0/3497823247 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76ac0263a0 con 0x7f76bc0f5410 2024-01-16T17:18:27.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.659+0000 7f76b27fc700 1 --2- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f76a802ffb0 0x7f76a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:27.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.659+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc045ad0 con 0x7f76bc0f5410 2024-01-16T17:18:27.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.659+0000 7f76b27fc700 1 -- 172.21.15.190:0/3497823247 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76ac0410b0 con 0x7f76bc0f5410 2024-01-16T17:18:27.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.659+0000 7f76c12cf700 1 --2- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f76a802ffb0 0x7f76a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:27.639 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.660+0000 7f76c12cf700 1 --2- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f76a802ffb0 0x7f76a8032470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f76b80096a0 tx=0x7f76b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:27.642 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.664+0000 7f76b27fc700 1 -- 172.21.15.190:0/3497823247 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76ac012920 con 0x7f76bc0f5410 2024-01-16T17:18:27.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.869+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76bc1080d0 con 0x7f76bc0f5410 2024-01-16T17:18:27.848 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.870+0000 7f76b27fc700 1 -- 172.21.15.190:0/3497823247 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76ac02b020 con 0x7f76bc0f5410 2024-01-16T17:18:27.848 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:27.848 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f76a802ffb0 msgr2=0x7f76a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 --2- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f76a802ffb0 0x7f76a8032470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f76b80096a0 tx=0x7f76b8006b40).stop 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 msgr2=0x7f76bc106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 --2- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 0x7f76bc106710 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f76ac013510 tx=0x7f76ac00b040).stop 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 shutdown_connections 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 --2- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f76a802ffb0 0x7f76a8032470 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 --2- 172.21.15.190:0/3497823247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f76bc0f5410 0x7f76bc106710 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 >> 172.21.15.190:0/3497823247 conn(0x7f76bc0f0b70 msgr2=0x7f76bc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 shutdown_connections 2024-01-16T17:18:27.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:27.872+0000 7f76c3d34700 1 -- 172.21.15.190:0/3497823247 wait complete. 2024-01-16T17:18:27.851 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:27.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:25.855746+0000 mgr.smithi154.sllivz (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:28.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:28 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:27.871120+0000 mon.smithi154 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.190:0/3497823247' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:29.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:29.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:29.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:27.856022+0000 mgr.smithi154.sllivz (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:30.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.316+0000 7f84e137d700 1 -- 172.21.15.190:0/706588342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 msgr2=0x7f84dc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:30.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.316+0000 7f84e137d700 1 --2- 172.21.15.190:0/706588342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 0x7f84dc0f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f84cc009fb0 tx=0x7f84cc009b30).stop 2024-01-16T17:18:30.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.316+0000 7f84e137d700 1 -- 172.21.15.190:0/706588342 shutdown_connections 2024-01-16T17:18:30.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.316+0000 7f84e137d700 1 --2- 172.21.15.190:0/706588342 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 0x7f84dc0f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:30.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.316+0000 7f84e137d700 1 -- 172.21.15.190:0/706588342 >> 172.21.15.190:0/706588342 conn(0x7f84dc0f1c70 msgr2=0x7f84dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:30.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.316+0000 7f84e137d700 1 -- 172.21.15.190:0/706588342 shutdown_connections 2024-01-16T17:18:30.294 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.316+0000 7f84e137d700 1 -- 172.21.15.190:0/706588342 wait complete. 2024-01-16T17:18:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.316+0000 7f84e137d700 1 Processor -- start 2024-01-16T17:18:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84e137d700 1 -- start start 2024-01-16T17:18:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84e137d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 0x7f84dc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84e137d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc106c20 con 0x7f84dc0f6510 2024-01-16T17:18:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84daffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 0x7f84dc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84daffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 0x7f84dc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58076/0 (socket says 172.21.15.190:58076) 2024-01-16T17:18:30.296 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84daffd700 1 -- 172.21.15.190:0/2754217356 learned_addr learned my addr 172.21.15.190:0/2754217356 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:30.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84daffd700 1 -- 172.21.15.190:0/2754217356 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc005040 con 0x7f84dc0f6510 2024-01-16T17:18:30.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84daffd700 1 --2- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 0x7f84dc106730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f84cc013510 tx=0x7f84cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:30.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.317+0000 7f84c37fe700 1 -- 172.21.15.190:0/2754217356 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84cc009320 con 0x7f84dc0f6510 2024-01-16T17:18:30.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.318+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc106e20 con 0x7f84dc0f6510 2024-01-16T17:18:30.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.318+0000 7f84c37fe700 1 -- 172.21.15.190:0/2754217356 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84cc013f00 con 0x7f84dc0f6510 2024-01-16T17:18:30.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.318+0000 7f84c37fe700 1 -- 172.21.15.190:0/2754217356 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84cc014d90 con 0x7f84dc0f6510 2024-01-16T17:18:30.297 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.318+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc1072b0 con 0x7f84dc0f6510 2024-01-16T17:18:30.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.318+0000 7f84c37fe700 1 -- 172.21.15.190:0/2754217356 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84cc014420 con 0x7f84dc0f6510 2024-01-16T17:18:30.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.319+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc045ad0 con 0x7f84dc0f6510 2024-01-16T17:18:30.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.319+0000 7f84c37fe700 1 --2- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:30.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.319+0000 7f84c37fe700 1 -- 172.21.15.190:0/2754217356 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84cc041050 con 0x7f84dc0f6510 2024-01-16T17:18:30.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.319+0000 7f84da7fc700 1 --2- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:30.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.319+0000 7f84da7fc700 1 --2- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f84d00096a0 tx=0x7f84d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:30.302 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.324+0000 7f84c37fe700 1 -- 172.21.15.190:0/2754217356 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84cc0095f0 con 0x7f84dc0f6510 2024-01-16T17:18:30.509 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.531+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84dc0fbf60 con 0x7f84dc0f6510 2024-01-16T17:18:30.509 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.532+0000 7f84c37fe700 1 -- 172.21.15.190:0/2754217356 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f84cc01db70 con 0x7f84dc0f6510 2024-01-16T17:18:30.510 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:30.510 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:30.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 msgr2=0x7f84c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:30.511 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 --2- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f84d00096a0 tx=0x7f84d0006b40).stop 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 msgr2=0x7f84dc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 --2- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 0x7f84dc106730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f84cc013510 tx=0x7f84cc00b040).stop 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 shutdown_connections 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 --2- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f84c402ff60 0x7f84c4032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 --2- 172.21.15.190:0/2754217356 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84dc0f6510 0x7f84dc106730 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 >> 172.21.15.190:0/2754217356 conn(0x7f84dc0f1c70 msgr2=0x7f84dc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 shutdown_connections 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:30.533+0000 7f84e137d700 1 -- 172.21.15.190:0/2754217356 wait complete. 2024-01-16T17:18:30.512 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:30.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:30 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:30.532811+0000 mon.smithi154 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.190:0/2754217356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:31.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:31.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:31.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:29.856378+0000 mgr.smithi154.sllivz (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:32.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.988+0000 7fe818ece700 1 -- 172.21.15.190:0/1849838475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 msgr2=0x7fe8140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:32.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.988+0000 7fe818ece700 1 --2- 172.21.15.190:0/1849838475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 0x7fe8140f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fe804009fb0 tx=0x7fe804009b30).stop 2024-01-16T17:18:32.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.988+0000 7fe818ece700 1 -- 172.21.15.190:0/1849838475 shutdown_connections 2024-01-16T17:18:32.969 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.988+0000 7fe818ece700 1 --2- 172.21.15.190:0/1849838475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 0x7fe8140f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.988+0000 7fe818ece700 1 -- 172.21.15.190:0/1849838475 >> 172.21.15.190:0/1849838475 conn(0x7fe8140f0b70 msgr2=0x7fe8140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.988+0000 7fe818ece700 1 -- 172.21.15.190:0/1849838475 shutdown_connections 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.988+0000 7fe818ece700 1 -- 172.21.15.190:0/1849838475 wait complete. 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.989+0000 7fe818ece700 1 Processor -- start 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.989+0000 7fe818ece700 1 -- start start 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.989+0000 7fe818ece700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 0x7fe8141093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.989+0000 7fe818ece700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe814060280 con 0x7fe8140f5410 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.989+0000 7fe81259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 0x7fe8141093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe81259c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 0x7fe8141093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58212/0 (socket says 172.21.15.190:58212) 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe81259c700 1 -- 172.21.15.190:0/237717897 learned_addr learned my addr 172.21.15.190:0/237717897 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe81259c700 1 -- 172.21.15.190:0/237717897 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe804005040 con 0x7fe8140f5410 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe81259c700 1 --2- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 0x7fe8141093f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fe804013510 tx=0x7fe80400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe80affd700 1 -- 172.21.15.190:0/237717897 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe804009320 con 0x7fe8140f5410 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe814060480 con 0x7fe8140f5410 2024-01-16T17:18:32.970 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe80affd700 1 -- 172.21.15.190:0/237717897 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe804013f00 con 0x7fe8140f5410 2024-01-16T17:18:32.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe80affd700 1 -- 172.21.15.190:0/237717897 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe804014d90 con 0x7fe8140f5410 2024-01-16T17:18:32.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.990+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe814109b20 con 0x7fe8140f5410 2024-01-16T17:18:32.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.991+0000 7fe80affd700 1 -- 172.21.15.190:0/237717897 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe804014420 con 0x7fe8140f5410 2024-01-16T17:18:32.971 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.991+0000 7fe80affd700 1 --2- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe80002ffb0 0x7fe800032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:32.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.991+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe814045ad0 con 0x7fe8140f5410 2024-01-16T17:18:32.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.991+0000 7fe80affd700 1 -- 172.21.15.190:0/237717897 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe804041070 con 0x7fe8140f5410 2024-01-16T17:18:32.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.991+0000 7fe811d9b700 1 --2- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe80002ffb0 0x7fe800032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:32.972 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.992+0000 7fe811d9b700 1 --2- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe80002ffb0 0x7fe800032470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fe7fc0096a0 tx=0x7fe7fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:32.974 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:32.996+0000 7fe80affd700 1 -- 172.21.15.190:0/237717897 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8040095f0 con 0x7fe8140f5410 2024-01-16T17:18:33.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.202+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe81410a340 con 0x7fe8140f5410 2024-01-16T17:18:33.181 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.202+0000 7fe80affd700 1 -- 172.21.15.190:0/237717897 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe80401db70 con 0x7fe8140f5410 2024-01-16T17:18:33.181 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:33.181 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:33.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe80002ffb0 msgr2=0x7fe800032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:33.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 --2- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe80002ffb0 0x7fe800032470 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fe7fc0096a0 tx=0x7fe7fc006b40).stop 2024-01-16T17:18:33.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 msgr2=0x7fe8141093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:33.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 --2- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 0x7fe8141093f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fe804013510 tx=0x7fe80400b040).stop 2024-01-16T17:18:33.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 shutdown_connections 2024-01-16T17:18:33.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 --2- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fe80002ffb0 0x7fe800032470 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:33.183 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 --2- 172.21.15.190:0/237717897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8140f5410 0x7fe8141093f0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:33.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 >> 172.21.15.190:0/237717897 conn(0x7fe8140f0b70 msgr2=0x7fe8140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:33.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 shutdown_connections 2024-01-16T17:18:33.184 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:33.204+0000 7fe818ece700 1 -- 172.21.15.190:0/237717897 wait complete. 2024-01-16T17:18:33.184 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:33.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:31.856697+0000 mgr.smithi154.sllivz (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:33.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:33.204415+0000 mon.smithi154 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.190:0/237717897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:34.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:34.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:35.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.814+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1213687889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 msgr2=0x7fed140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:35.795 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.814+0000 7fed1ab7d700 1 --2- 172.21.15.190:0/1213687889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 0x7fed140f6930 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fed08009fb0 tx=0x7fed08009b30).stop 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.814+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1213687889 shutdown_connections 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.814+0000 7fed1ab7d700 1 --2- 172.21.15.190:0/1213687889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 0x7fed140f6930 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.814+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1213687889 >> 172.21.15.190:0/1213687889 conn(0x7fed140f1c70 msgr2=0x7fed140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.814+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1213687889 shutdown_connections 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.814+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1213687889 wait complete. 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.815+0000 7fed1ab7d700 1 Processor -- start 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.815+0000 7fed1ab7d700 1 -- start start 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.815+0000 7fed1ab7d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 0x7fed14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.815+0000 7fed1ab7d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed14106c20 con 0x7fed140f6510 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed18919700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 0x7fed14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed18919700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 0x7fed14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58222/0 (socket says 172.21.15.190:58222) 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed18919700 1 -- 172.21.15.190:0/1934547939 learned_addr learned my addr 172.21.15.190:0/1934547939 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed18919700 1 -- 172.21.15.190:0/1934547939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed08005040 con 0x7fed140f6510 2024-01-16T17:18:35.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed18919700 1 --2- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 0x7fed14106730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fed08014000 tx=0x7fed0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:35.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed117fa700 1 -- 172.21.15.190:0/1934547939 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed080095f0 con 0x7fed140f6510 2024-01-16T17:18:35.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed14106e20 con 0x7fed140f6510 2024-01-16T17:18:35.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed117fa700 1 -- 172.21.15.190:0/1934547939 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed08007e70 con 0x7fed140f6510 2024-01-16T17:18:35.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed117fa700 1 -- 172.21.15.190:0/1934547939 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed08014ea0 con 0x7fed140f6510 2024-01-16T17:18:35.797 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.816+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed141072b0 con 0x7fed140f6510 2024-01-16T17:18:35.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.817+0000 7fed117fa700 1 -- 172.21.15.190:0/1934547939 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed08025070 con 0x7fed140f6510 2024-01-16T17:18:35.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.817+0000 7fed117fa700 1 --2- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed0402ff60 0x7fed04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:35.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.817+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed14045ad0 con 0x7fed140f6510 2024-01-16T17:18:35.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.817+0000 7fed117fa700 1 -- 172.21.15.190:0/1934547939 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed080409f0 con 0x7fed140f6510 2024-01-16T17:18:35.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.818+0000 7fed13fff700 1 --2- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed0402ff60 0x7fed04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:35.798 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.818+0000 7fed13fff700 1 --2- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed0402ff60 0x7fed04032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fed000096a0 tx=0x7fed00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:35.801 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:35.822+0000 7fed117fa700 1 -- 172.21.15.190:0/1934547939 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed0800f4e0 con 0x7fed140f6510 2024-01-16T17:18:35.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:33.857046+0000 mgr.smithi154.sllivz (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:36.010 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.031+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed140fc050 con 0x7fed140f6510 2024-01-16T17:18:36.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.032+0000 7fed117fa700 1 -- 172.21.15.190:0/1934547939 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed0800f300 con 0x7fed140f6510 2024-01-16T17:18:36.011 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:36.011 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed0402ff60 msgr2=0x7fed04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 --2- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed0402ff60 0x7fed04032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fed000096a0 tx=0x7fed00006b40).stop 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 msgr2=0x7fed14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 --2- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 0x7fed14106730 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fed08014000 tx=0x7fed0800b040).stop 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 shutdown_connections 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 --2- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fed0402ff60 0x7fed04032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 --2- 172.21.15.190:0/1934547939 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed140f6510 0x7fed14106730 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 >> 172.21.15.190:0/1934547939 conn(0x7fed140f1c70 msgr2=0x7fed140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 shutdown_connections 2024-01-16T17:18:36.013 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:36.034+0000 7fed1ab7d700 1 -- 172.21.15.190:0/1934547939 wait complete. 2024-01-16T17:18:36.014 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:36.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:36.034072+0000 mon.smithi154 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.190:0/1934547939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:37.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:37.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:37.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:35.857314+0000 mgr.smithi154.sllivz (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:38.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 -- 172.21.15.190:0/1449964076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 msgr2=0x7fda300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:38.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 --2- 172.21.15.190:0/1449964076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 0x7fda300f66c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fda20009fb0 tx=0x7fda20009b30).stop 2024-01-16T17:18:38.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 -- 172.21.15.190:0/1449964076 shutdown_connections 2024-01-16T17:18:38.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 --2- 172.21.15.190:0/1449964076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 0x7fda300f66c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:38.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 -- 172.21.15.190:0/1449964076 >> 172.21.15.190:0/1449964076 conn(0x7fda300f1aa0 msgr2=0x7fda300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:38.673 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 -- 172.21.15.190:0/1449964076 shutdown_connections 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 -- 172.21.15.190:0/1449964076 wait complete. 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 Processor -- start 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 -- start start 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 0x7fda301064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.693+0000 7fda366a7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda301069a0 con 0x7fda300f62a0 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda356a5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 0x7fda301064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda356a5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 0x7fda301064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58240/0 (socket says 172.21.15.190:58240) 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda356a5700 1 -- 172.21.15.190:0/692570865 learned_addr learned my addr 172.21.15.190:0/692570865 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda356a5700 1 -- 172.21.15.190:0/692570865 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20005040 con 0x7fda300f62a0 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda356a5700 1 --2- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 0x7fda301064b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fda20013510 tx=0x7fda2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda25ffb700 1 -- 172.21.15.190:0/692570865 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda20009320 con 0x7fda300f62a0 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda30106ba0 con 0x7fda300f62a0 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda25ffb700 1 -- 172.21.15.190:0/692570865 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda20013f00 con 0x7fda300f62a0 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda25ffb700 1 -- 172.21.15.190:0/692570865 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda20014dc0 con 0x7fda300f62a0 2024-01-16T17:18:38.674 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.694+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda30107030 con 0x7fda300f62a0 2024-01-16T17:18:38.675 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.695+0000 7fda25ffb700 1 -- 172.21.15.190:0/692570865 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda2001d3d0 con 0x7fda300f62a0 2024-01-16T17:18:38.675 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.695+0000 7fda25ffb700 1 --2- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda1c02ff60 0x7fda1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:38.675 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.695+0000 7fda25ffb700 1 -- 172.21.15.190:0/692570865 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda20041010 con 0x7fda300f62a0 2024-01-16T17:18:38.675 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.695+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda30045ad0 con 0x7fda300f62a0 2024-01-16T17:18:38.676 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.695+0000 7fda34ea4700 1 --2- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda1c02ff60 0x7fda1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:38.676 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.696+0000 7fda34ea4700 1 --2- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda1c02ff60 0x7fda1c032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fda2c0096a0 tx=0x7fda2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:38.679 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.701+0000 7fda25ffb700 1 -- 172.21.15.190:0/692570865 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda20012920 con 0x7fda300f62a0 2024-01-16T17:18:38.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.904+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda30107d70 con 0x7fda300f62a0 2024-01-16T17:18:38.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.905+0000 7fda25ffb700 1 -- 172.21.15.190:0/692570865 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda20012330 con 0x7fda300f62a0 2024-01-16T17:18:38.884 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:38.884 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:38.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda1c02ff60 msgr2=0x7fda1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:38.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 --2- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda1c02ff60 0x7fda1c032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fda2c0096a0 tx=0x7fda2c006b40).stop 2024-01-16T17:18:38.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 msgr2=0x7fda301064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:38.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 --2- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 0x7fda301064b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fda20013510 tx=0x7fda2000b040).stop 2024-01-16T17:18:38.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 shutdown_connections 2024-01-16T17:18:38.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 --2- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fda1c02ff60 0x7fda1c032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:38.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 --2- 172.21.15.190:0/692570865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda300f62a0 0x7fda301064b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:38.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 >> 172.21.15.190:0/692570865 conn(0x7fda300f1aa0 msgr2=0x7fda300f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 shutdown_connections 2024-01-16T17:18:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:38.907+0000 7fda366a7700 1 -- 172.21.15.190:0/692570865 wait complete. 2024-01-16T17:18:38.887 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:39.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:37.857596+0000 mgr.smithi154.sllivz (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:39.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:38.907264+0000 mon.smithi154 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.190:0/692570865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:40.246 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:40.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:41.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 -- 172.21.15.190:0/3935654466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 msgr2=0x7f50c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:41.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 --2- 172.21.15.190:0/3935654466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 0x7f50c80f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f50b8009fb0 tx=0x7f50b8009b30).stop 2024-01-16T17:18:41.402 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 -- 172.21.15.190:0/3935654466 shutdown_connections 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 --2- 172.21.15.190:0/3935654466 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 0x7f50c80f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 -- 172.21.15.190:0/3935654466 >> 172.21.15.190:0/3935654466 conn(0x7f50c80f0b70 msgr2=0x7f50c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 -- 172.21.15.190:0/3935654466 shutdown_connections 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 -- 172.21.15.190:0/3935654466 wait complete. 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 Processor -- start 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 -- start start 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 0x7f50c8108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.423+0000 7f50cca2e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c8108e10 con 0x7f50c80f5410 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50c659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 0x7f50c8108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50c659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 0x7f50c8108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58268/0 (socket says 172.21.15.190:58268) 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50c659c700 1 -- 172.21.15.190:0/1209968482 learned_addr learned my addr 172.21.15.190:0/1209968482 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50c659c700 1 -- 172.21.15.190:0/1209968482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b8005040 con 0x7f50c80f5410 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50c659c700 1 --2- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 0x7f50c8108920 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f50b8013510 tx=0x7f50b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50b6ffd700 1 -- 172.21.15.190:0/1209968482 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b8009320 con 0x7f50c80f5410 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c8109010 con 0x7f50c80f5410 2024-01-16T17:18:41.403 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50b6ffd700 1 -- 172.21.15.190:0/1209968482 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50b8013f00 con 0x7f50c80f5410 2024-01-16T17:18:41.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50b6ffd700 1 -- 172.21.15.190:0/1209968482 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b8014d90 con 0x7f50c80f5410 2024-01-16T17:18:41.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.424+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c81094a0 con 0x7f50c80f5410 2024-01-16T17:18:41.404 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.425+0000 7f50b6ffd700 1 -- 172.21.15.190:0/1209968482 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50b80095f0 con 0x7f50c80f5410 2024-01-16T17:18:41.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.425+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c8045ad0 con 0x7f50c80f5410 2024-01-16T17:18:41.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.425+0000 7f50b6ffd700 1 --2- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f50b002ff60 0x7f50b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:41.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.425+0000 7f50b6ffd700 1 -- 172.21.15.190:0/1209968482 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50b80410c0 con 0x7f50c80f5410 2024-01-16T17:18:41.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.425+0000 7f50c5d9b700 1 --2- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f50b002ff60 0x7f50b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:41.405 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.426+0000 7f50c5d9b700 1 --2- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f50b002ff60 0x7f50b0032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f50bc0096a0 tx=0x7f50bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:41.408 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.430+0000 7f50b6ffd700 1 -- 172.21.15.190:0/1209968482 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50b8014420 con 0x7f50c80f5410 2024-01-16T17:18:41.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.638+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50c810a350 con 0x7f50c80f5410 2024-01-16T17:18:41.616 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.639+0000 7f50b6ffd700 1 -- 172.21.15.190:0/1209968482 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f50b801d7f0 con 0x7f50c80f5410 2024-01-16T17:18:41.617 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:41.617 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:41.618 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f50b002ff60 msgr2=0x7f50b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:41.618 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 --2- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f50b002ff60 0x7f50b0032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f50bc0096a0 tx=0x7f50bc006b40).stop 2024-01-16T17:18:41.618 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 msgr2=0x7f50c8108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:41.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 --2- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 0x7f50c8108920 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f50b8013510 tx=0x7f50b800b040).stop 2024-01-16T17:18:41.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 shutdown_connections 2024-01-16T17:18:41.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 --2- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f50b002ff60 0x7f50b0032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:41.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 --2- 172.21.15.190:0/1209968482 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c80f5410 0x7f50c8108920 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:41.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 >> 172.21.15.190:0/1209968482 conn(0x7f50c80f0b70 msgr2=0x7f50c805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:41.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 shutdown_connections 2024-01-16T17:18:41.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:41.640+0000 7f50cca2e700 1 -- 172.21.15.190:0/1209968482 wait complete. 2024-01-16T17:18:41.619 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:41.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:39.857887+0000 mgr.smithi154.sllivz (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:42.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:42.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:42.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:42 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:41.639939+0000 mon.smithi154 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.190:0/1209968482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:43.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:41.858209+0000 mgr.smithi154.sllivz (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:44.058 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.079+0000 7fba254c1700 1 -- 172.21.15.190:0/1636796183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 msgr2=0x7fba200f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.079+0000 7fba254c1700 1 --2- 172.21.15.190:0/1636796183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 0x7fba200f4be0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fba14009fb0 tx=0x7fba14009b30).stop 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.079+0000 7fba254c1700 1 -- 172.21.15.190:0/1636796183 shutdown_connections 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.079+0000 7fba254c1700 1 --2- 172.21.15.190:0/1636796183 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 0x7fba200f4be0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.079+0000 7fba254c1700 1 -- 172.21.15.190:0/1636796183 >> 172.21.15.190:0/1636796183 conn(0x7fba200f02e0 msgr2=0x7fba200f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.079+0000 7fba254c1700 1 -- 172.21.15.190:0/1636796183 shutdown_connections 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.079+0000 7fba254c1700 1 -- 172.21.15.190:0/1636796183 wait complete. 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba254c1700 1 Processor -- start 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba254c1700 1 -- start start 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba254c1700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 0x7fba200f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba254c1700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba200f57f0 con 0x7fba200f47c0 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba1effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 0x7fba200f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba1effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 0x7fba200f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:41960/0 (socket says 172.21.15.190:41960) 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba1effd700 1 -- 172.21.15.190:0/1432934986 learned_addr learned my addr 172.21.15.190:0/1432934986 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba1effd700 1 -- 172.21.15.190:0/1432934986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba14005040 con 0x7fba200f47c0 2024-01-16T17:18:44.059 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.080+0000 7fba1effd700 1 --2- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 0x7fba200f7370 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fba14013510 tx=0x7fba1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:44.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.081+0000 7fba077fe700 1 -- 172.21.15.190:0/1432934986 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba14009320 con 0x7fba200f47c0 2024-01-16T17:18:44.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.081+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba200f5990 con 0x7fba200f47c0 2024-01-16T17:18:44.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.081+0000 7fba077fe700 1 -- 172.21.15.190:0/1432934986 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba14013f00 con 0x7fba200f47c0 2024-01-16T17:18:44.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.081+0000 7fba077fe700 1 -- 172.21.15.190:0/1432934986 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba14014d90 con 0x7fba200f47c0 2024-01-16T17:18:44.060 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.081+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba200f5e20 con 0x7fba200f47c0 2024-01-16T17:18:44.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.081+0000 7fba077fe700 1 -- 172.21.15.190:0/1432934986 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba140263a0 con 0x7fba200f47c0 2024-01-16T17:18:44.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.081+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba20059100 con 0x7fba200f47c0 2024-01-16T17:18:44.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.082+0000 7fba077fe700 1 --2- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba0802ff60 0x7fba08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:44.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.082+0000 7fba077fe700 1 -- 172.21.15.190:0/1432934986 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba14041000 con 0x7fba200f47c0 2024-01-16T17:18:44.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.082+0000 7fba1e7fc700 1 --2- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba0802ff60 0x7fba08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:44.061 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.082+0000 7fba1e7fc700 1 --2- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba0802ff60 0x7fba08032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fba100096a0 tx=0x7fba10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:44.064 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.086+0000 7fba077fe700 1 -- 172.21.15.190:0/1432934986 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba140095f0 con 0x7fba200f47c0 2024-01-16T17:18:44.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.293+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba200fbf60 con 0x7fba200f47c0 2024-01-16T17:18:44.271 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.293+0000 7fba077fe700 1 -- 172.21.15.190:0/1432934986 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba1401db70 con 0x7fba200f47c0 2024-01-16T17:18:44.271 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:44.272 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:44.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba0802ff60 msgr2=0x7fba08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:44.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 --2- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba0802ff60 0x7fba08032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fba100096a0 tx=0x7fba10006b40).stop 2024-01-16T17:18:44.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 msgr2=0x7fba200f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:44.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 --2- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 0x7fba200f7370 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fba14013510 tx=0x7fba1400b040).stop 2024-01-16T17:18:44.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 shutdown_connections 2024-01-16T17:18:44.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 --2- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba0802ff60 0x7fba08032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:44.273 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 --2- 172.21.15.190:0/1432934986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba200f47c0 0x7fba200f7370 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:44.274 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 >> 172.21.15.190:0/1432934986 conn(0x7fba200f02e0 msgr2=0x7fba200f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:44.274 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 shutdown_connections 2024-01-16T17:18:44.274 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:44.295+0000 7fba254c1700 1 -- 172.21.15.190:0/1432934986 wait complete. 2024-01-16T17:18:44.274 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:44.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:44 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:44.294612+0000 mon.smithi154 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.190:0/1432934986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:45.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:45.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:45.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:43.858607+0000 mgr.smithi154.sllivz (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.794+0000 7f43b9392700 1 -- 172.21.15.190:0/3040474233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 msgr2=0x7f43b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.794+0000 7f43b9392700 1 --2- 172.21.15.190:0/3040474233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 0x7f43b40f6930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f43a8009fb0 tx=0x7f43a8009b30).stop 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.794+0000 7f43b9392700 1 -- 172.21.15.190:0/3040474233 shutdown_connections 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.794+0000 7f43b9392700 1 --2- 172.21.15.190:0/3040474233 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 0x7f43b40f6930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.794+0000 7f43b9392700 1 -- 172.21.15.190:0/3040474233 >> 172.21.15.190:0/3040474233 conn(0x7f43b40f1c70 msgr2=0x7f43b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.794+0000 7f43b9392700 1 -- 172.21.15.190:0/3040474233 shutdown_connections 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.794+0000 7f43b9392700 1 -- 172.21.15.190:0/3040474233 wait complete. 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.795+0000 7f43b9392700 1 Processor -- start 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.795+0000 7f43b9392700 1 -- start start 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.795+0000 7f43b9392700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 0x7f43b406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.795+0000 7f43b9392700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b4069e20 con 0x7f43b40f6510 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.795+0000 7f43b2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 0x7f43b406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:46.774 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.795+0000 7f43b2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 0x7f43b406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:41988/0 (socket says 172.21.15.190:41988) 2024-01-16T17:18:46.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.795+0000 7f43b2ffd700 1 -- 172.21.15.190:0/4022638494 learned_addr learned my addr 172.21.15.190:0/4022638494 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:46.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.796+0000 7f43b2ffd700 1 -- 172.21.15.190:0/4022638494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a8005040 con 0x7f43b40f6510 2024-01-16T17:18:46.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.796+0000 7f43b2ffd700 1 --2- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 0x7f43b406b9a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f43a8013510 tx=0x7f43a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:46.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.796+0000 7f439b7fe700 1 -- 172.21.15.190:0/4022638494 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a8009320 con 0x7f43b40f6510 2024-01-16T17:18:46.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.796+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b4069fc0 con 0x7f43b40f6510 2024-01-16T17:18:46.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.796+0000 7f439b7fe700 1 -- 172.21.15.190:0/4022638494 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43a8013f00 con 0x7f43b40f6510 2024-01-16T17:18:46.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.796+0000 7f439b7fe700 1 -- 172.21.15.190:0/4022638494 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a8014d90 con 0x7f43b40f6510 2024-01-16T17:18:46.775 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.796+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b406a450 con 0x7f43b40f6510 2024-01-16T17:18:46.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.797+0000 7f439b7fe700 1 -- 172.21.15.190:0/4022638494 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43a80263a0 con 0x7f43b40f6510 2024-01-16T17:18:46.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.797+0000 7f439b7fe700 1 --2- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f439c02ffb0 0x7f439c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:46.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.797+0000 7f439b7fe700 1 -- 172.21.15.190:0/4022638494 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43a8041100 con 0x7f43b40f6510 2024-01-16T17:18:46.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.797+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b4059100 con 0x7f43b40f6510 2024-01-16T17:18:46.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.797+0000 7f43b27fc700 1 --2- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f439c02ffb0 0x7f439c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:46.776 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.797+0000 7f43b27fc700 1 --2- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f439c02ffb0 0x7f439c032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f43a40096a0 tx=0x7f43a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:46.780 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:46.802+0000 7f439b7fe700 1 -- 172.21.15.190:0/4022638494 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43a8012920 con 0x7f43b40f6510 2024-01-16T17:18:46.982 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.005+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43b406aab0 con 0x7f43b40f6510 2024-01-16T17:18:46.983 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.005+0000 7f439b7fe700 1 -- 172.21.15.190:0/4022638494 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f43a802b020 con 0x7f43b40f6510 2024-01-16T17:18:46.984 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:46.984 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:46.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f439c02ffb0 msgr2=0x7f439c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:46.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 --2- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f439c02ffb0 0x7f439c032470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f43a40096a0 tx=0x7f43a4006b40).stop 2024-01-16T17:18:46.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 msgr2=0x7f43b406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:46.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 --2- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 0x7f43b406b9a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f43a8013510 tx=0x7f43a800b040).stop 2024-01-16T17:18:46.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 shutdown_connections 2024-01-16T17:18:46.985 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 --2- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f439c02ffb0 0x7f439c032470 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:46.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 --2- 172.21.15.190:0/4022638494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43b40f6510 0x7f43b406b9a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:46.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 >> 172.21.15.190:0/4022638494 conn(0x7f43b40f1c70 msgr2=0x7f43b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:46.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 shutdown_connections 2024-01-16T17:18:46.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:47.007+0000 7f43b9392700 1 -- 172.21.15.190:0/4022638494 wait complete. 2024-01-16T17:18:46.986 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:47.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:45.858982+0000 mgr.smithi154.sllivz (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:47.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:47.006391+0000 mon.smithi154 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.190:0/4022638494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:48.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:48.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.548+0000 7f0c71f44700 1 -- 172.21.15.190:0/2273471667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 msgr2=0x7f0c6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.548+0000 7f0c71f44700 1 --2- 172.21.15.190:0/2273471667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c0f6930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f0c60009fb0 tx=0x7f0c60009b30).stop 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.548+0000 7f0c71f44700 1 -- 172.21.15.190:0/2273471667 shutdown_connections 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.548+0000 7f0c71f44700 1 --2- 172.21.15.190:0/2273471667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c0f6930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.548+0000 7f0c71f44700 1 -- 172.21.15.190:0/2273471667 >> 172.21.15.190:0/2273471667 conn(0x7f0c6c0f1c70 msgr2=0x7f0c6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.548+0000 7f0c71f44700 1 -- 172.21.15.190:0/2273471667 shutdown_connections 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.548+0000 7f0c71f44700 1 -- 172.21.15.190:0/2273471667 wait complete. 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.549+0000 7f0c71f44700 1 Processor -- start 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.549+0000 7f0c71f44700 1 -- start start 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.549+0000 7f0c71f44700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:49.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.549+0000 7f0c71f44700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c069e20 con 0x7f0c6c0f6510 2024-01-16T17:18:49.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.549+0000 7f0c6b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:49.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.549+0000 7f0c6b7fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:42000/0 (socket says 172.21.15.190:42000) 2024-01-16T17:18:49.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.549+0000 7f0c6b7fe700 1 -- 172.21.15.190:0/1956749225 learned_addr learned my addr 172.21.15.190:0/1956749225 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:49.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.550+0000 7f0c6b7fe700 1 -- 172.21.15.190:0/1956749225 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c60005040 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.550+0000 7f0c6b7fe700 1 --2- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c06b9a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f0c60013510 tx=0x7f0c6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.550+0000 7f0c53fff700 1 -- 172.21.15.190:0/1956749225 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c60009320 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.550+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c069fc0 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.550+0000 7f0c53fff700 1 -- 172.21.15.190:0/1956749225 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c60013f00 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.550+0000 7f0c53fff700 1 -- 172.21.15.190:0/1956749225 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c60014d90 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.550+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c06a450 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.551+0000 7f0c53fff700 1 -- 172.21.15.190:0/1956749225 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c60014420 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.551+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c059100 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.551+0000 7f0c53fff700 1 --2- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c5402ff60 0x7f0c54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.551+0000 7f0c53fff700 1 -- 172.21.15.190:0/1956749225 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c600410c0 con 0x7f0c6c0f6510 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.551+0000 7f0c6affd700 1 --2- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c5402ff60 0x7f0c54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:49.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.551+0000 7f0c6affd700 1 --2- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c5402ff60 0x7f0c54032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0c5c0096a0 tx=0x7f0c5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:49.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.555+0000 7f0c53fff700 1 -- 172.21.15.190:0/1956749225 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c600095f0 con 0x7f0c6c0f6510 2024-01-16T17:18:49.741 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.763+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c6c0fbf60 con 0x7f0c6c0f6510 2024-01-16T17:18:49.742 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.764+0000 7f0c53fff700 1 -- 172.21.15.190:0/1956749225 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0c6001db70 con 0x7f0c6c0f6510 2024-01-16T17:18:49.742 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:49.742 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:49.743 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.765+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c5402ff60 msgr2=0x7f0c54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.765+0000 7f0c71f44700 1 --2- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c5402ff60 0x7f0c54032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0c5c0096a0 tx=0x7f0c5c006b40).stop 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.765+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 msgr2=0x7f0c6c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.765+0000 7f0c71f44700 1 --2- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c06b9a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f0c60013510 tx=0x7f0c6000b040).stop 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.765+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 shutdown_connections 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.765+0000 7f0c71f44700 1 --2- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0c5402ff60 0x7f0c54032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.765+0000 7f0c71f44700 1 --2- 172.21.15.190:0/1956749225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c6c0f6510 0x7f0c6c06b9a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.765+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 >> 172.21.15.190:0/1956749225 conn(0x7f0c6c0f1c70 msgr2=0x7f0c6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.766+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 shutdown_connections 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:49.766+0000 7f0c71f44700 1 -- 172.21.15.190:0/1956749225 wait complete. 2024-01-16T17:18:49.744 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:49.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:49 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:47.859289+0000 mgr.smithi154.sllivz (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:50.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:50 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:49.765095+0000 mon.smithi154 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.190:0/1956749225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:51.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:51.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:51.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:51 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:49.859560+0000 mgr.smithi154.sllivz (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:52.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.303+0000 7f0a829a7700 1 -- 172.21.15.190:0/535630095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 msgr2=0x7f0a7c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:52.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.303+0000 7f0a829a7700 1 --2- 172.21.15.190:0/535630095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 0x7f0a7c0f55c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f0a78009fb0 tx=0x7f0a78009b30).stop 2024-01-16T17:18:52.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.303+0000 7f0a829a7700 1 -- 172.21.15.190:0/535630095 shutdown_connections 2024-01-16T17:18:52.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.303+0000 7f0a829a7700 1 --2- 172.21.15.190:0/535630095 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 0x7f0a7c0f55c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:52.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.303+0000 7f0a829a7700 1 -- 172.21.15.190:0/535630095 >> 172.21.15.190:0/535630095 conn(0x7f0a7c0f09a0 msgr2=0x7f0a7c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:52.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.303+0000 7f0a829a7700 1 -- 172.21.15.190:0/535630095 shutdown_connections 2024-01-16T17:18:52.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.303+0000 7f0a829a7700 1 -- 172.21.15.190:0/535630095 wait complete. 2024-01-16T17:18:52.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.303+0000 7f0a829a7700 1 Processor -- start 2024-01-16T17:18:52.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a829a7700 1 -- start start 2024-01-16T17:18:52.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a829a7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 0x7f0a7c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:52.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a829a7700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c060240 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a819a5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 0x7f0a7c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:52.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a819a5700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 0x7f0a7c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36690/0 (socket says 172.21.15.190:36690) 2024-01-16T17:18:52.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a819a5700 1 -- 172.21.15.190:0/4128394032 learned_addr learned my addr 172.21.15.190:0/4128394032 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:52.284 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a819a5700 1 -- 172.21.15.190:0/4128394032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a78005040 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a819a5700 1 --2- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 0x7f0a7c1091b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0a78013510 tx=0x7f0a7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a727fc700 1 -- 172.21.15.190:0/4128394032 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a78009320 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c060440 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.304+0000 7f0a727fc700 1 -- 172.21.15.190:0/4128394032 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a78013f00 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.305+0000 7f0a727fc700 1 -- 172.21.15.190:0/4128394032 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a78014dc0 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.305+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c109860 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.305+0000 7f0a727fc700 1 -- 172.21.15.190:0/4128394032 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a780095f0 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.305+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c045ad0 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.305+0000 7f0a727fc700 1 --2- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0a6802ff60 0x7f0a68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:52.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.305+0000 7f0a727fc700 1 -- 172.21.15.190:0/4128394032 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a780410c0 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.306+0000 7f0a811a4700 1 --2- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0a6802ff60 0x7f0a68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:52.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.306+0000 7f0a811a4700 1 --2- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0a6802ff60 0x7f0a68032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f0a6c0096a0 tx=0x7f0a6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:52.288 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.310+0000 7f0a727fc700 1 -- 172.21.15.190:0/4128394032 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a7801c790 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.494 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.516+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a7c10a080 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.494 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.517+0000 7f0a727fc700 1 -- 172.21.15.190:0/4128394032 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0a78010230 con 0x7f0a7c0f51a0 2024-01-16T17:18:52.495 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:52.495 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:52.496 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0a6802ff60 msgr2=0x7f0a68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:52.496 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 --2- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0a6802ff60 0x7f0a68032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f0a6c0096a0 tx=0x7f0a6c006b40).stop 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 msgr2=0x7f0a7c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 --2- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 0x7f0a7c1091b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f0a78013510 tx=0x7f0a7800b040).stop 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 shutdown_connections 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 --2- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0a6802ff60 0x7f0a68032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 --2- 172.21.15.190:0/4128394032 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a7c0f51a0 0x7f0a7c1091b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 >> 172.21.15.190:0/4128394032 conn(0x7f0a7c0f09a0 msgr2=0x7f0a7c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 shutdown_connections 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:52.518+0000 7f0a829a7700 1 -- 172.21.15.190:0/4128394032 wait complete. 2024-01-16T17:18:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:52.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:52 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:52.517829+0000 mon.smithi154 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.190:0/4128394032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:53.706 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:53 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:51.859807+0000 mgr.smithi154.sllivz (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:53.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:53.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:54.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 -- 172.21.15.190:0/4217278786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 msgr2=0x7fd4a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:54.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 --2- 172.21.15.190:0/4217278786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 0x7fd4a80f5830 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fd490009fb0 tx=0x7fd490009b30).stop 2024-01-16T17:18:54.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 -- 172.21.15.190:0/4217278786 shutdown_connections 2024-01-16T17:18:54.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 --2- 172.21.15.190:0/4217278786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 0x7fd4a80f5830 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:54.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 -- 172.21.15.190:0/4217278786 >> 172.21.15.190:0/4217278786 conn(0x7fd4a80f0b70 msgr2=0x7fd4a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:54.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 -- 172.21.15.190:0/4217278786 shutdown_connections 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 -- 172.21.15.190:0/4217278786 wait complete. 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 Processor -- start 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 -- start start 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 0x7fd4a80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.978+0000 7fd4aca7f700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a80fb820 con 0x7fd4a80f5410 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd4a659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 0x7fd4a80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd4a659c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 0x7fd4a80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36702/0 (socket says 172.21.15.190:36702) 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd4a659c700 1 -- 172.21.15.190:0/1184004427 learned_addr learned my addr 172.21.15.190:0/1184004427 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd4a659c700 1 -- 172.21.15.190:0/1184004427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd490005040 con 0x7fd4a80f5410 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd4a659c700 1 --2- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 0x7fd4a80fd3a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd490014000 tx=0x7fd49000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd49effd700 1 -- 172.21.15.190:0/1184004427 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4900095f0 con 0x7fd4a80f5410 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd49effd700 1 -- 172.21.15.190:0/1184004427 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd490007e70 con 0x7fd4a80f5410 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a80fb9c0 con 0x7fd4a80f5410 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd49effd700 1 -- 172.21.15.190:0/1184004427 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd490014ea0 con 0x7fd4a80f5410 2024-01-16T17:18:54.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.979+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a80fbe50 con 0x7fd4a80f5410 2024-01-16T17:18:54.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.980+0000 7fd49effd700 1 -- 172.21.15.190:0/1184004427 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd490025070 con 0x7fd4a80f5410 2024-01-16T17:18:54.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.980+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a8059100 con 0x7fd4a80f5410 2024-01-16T17:18:54.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.980+0000 7fd49effd700 1 --2- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd49402ff60 0x7fd494032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:54.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.980+0000 7fd49effd700 1 -- 172.21.15.190:0/1184004427 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd490040970 con 0x7fd4a80f5410 2024-01-16T17:18:54.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.980+0000 7fd4a5d9b700 1 --2- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd49402ff60 0x7fd494032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:54.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.981+0000 7fd4a5d9b700 1 --2- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd49402ff60 0x7fd494032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd4980096a0 tx=0x7fd498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:54.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:54.985+0000 7fd49effd700 1 -- 172.21.15.190:0/1184004427 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4900079c0 con 0x7fd4a80f5410 2024-01-16T17:18:55.170 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.192+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4a80fae60 con 0x7fd4a80f5410 2024-01-16T17:18:55.171 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.193+0000 7fd49effd700 1 -- 172.21.15.190:0/1184004427 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd49000f300 con 0x7fd4a80f5410 2024-01-16T17:18:55.171 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:55.171 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd49402ff60 msgr2=0x7fd494032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 --2- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd49402ff60 0x7fd494032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd4980096a0 tx=0x7fd498006b40).stop 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 msgr2=0x7fd4a80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 --2- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 0x7fd4a80fd3a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd490014000 tx=0x7fd49000b040).stop 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 shutdown_connections 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 --2- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd49402ff60 0x7fd494032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 --2- 172.21.15.190:0/1184004427 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd4a80f5410 0x7fd4a80fd3a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 >> 172.21.15.190:0/1184004427 conn(0x7fd4a80f0b70 msgr2=0x7fd4a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 shutdown_connections 2024-01-16T17:18:55.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:55.195+0000 7fd4aca7f700 1 -- 172.21.15.190:0/1184004427 wait complete. 2024-01-16T17:18:55.174 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:55.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:53.860147+0000 mgr.smithi154.sllivz (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:55.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:55 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:55.194302+0000 mon.smithi154 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.190:0/1184004427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:56.547 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:56.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:56.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:55.932780+0000 mon.smithi154 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/mirror_snapshot_schedule"}]: dispatch 2024-01-16T17:18:56.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:56 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:55.933505+0000 mon.smithi154 (mon.0) 305 : audit [INF] from='mgr.14162 172.21.15.154:0/2275639842' entity='mgr.smithi154.sllivz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi154.sllivz/trash_purge_schedule"}]: dispatch 2024-01-16T17:18:57.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.738+0000 7f01539b6700 1 -- 172.21.15.190:0/694497120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 msgr2=0x7f014c0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:57.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.738+0000 7f01539b6700 1 --2- 172.21.15.190:0/694497120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 0x7f014c0f9190 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f013c009fb0 tx=0x7f013c009b30).stop 2024-01-16T17:18:57.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.738+0000 7f01539b6700 1 -- 172.21.15.190:0/694497120 shutdown_connections 2024-01-16T17:18:57.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.738+0000 7f01539b6700 1 --2- 172.21.15.190:0/694497120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 0x7f014c0f9190 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:57.718 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.738+0000 7f01539b6700 1 -- 172.21.15.190:0/694497120 >> 172.21.15.190:0/694497120 conn(0x7f014c0f0b70 msgr2=0x7f014c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.738+0000 7f01539b6700 1 -- 172.21.15.190:0/694497120 shutdown_connections 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.738+0000 7f01539b6700 1 -- 172.21.15.190:0/694497120 wait complete. 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.739+0000 7f01539b6700 1 Processor -- start 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.739+0000 7f01539b6700 1 -- start start 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.739+0000 7f01539b6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 0x7f014c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.739+0000 7f01539b6700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c106bd0 con 0x7f014c0601e0 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.739+0000 7f0151752700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 0x7f014c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f0151752700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 0x7f014c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36726/0 (socket says 172.21.15.190:36726) 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f0151752700 1 -- 172.21.15.190:0/2932426143 learned_addr learned my addr 172.21.15.190:0/2932426143 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f0151752700 1 -- 172.21.15.190:0/2932426143 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013c005040 con 0x7f014c0601e0 2024-01-16T17:18:57.719 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f0151752700 1 --2- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 0x7f014c1066e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f013c004dd0 tx=0x7f013c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:57.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f0141ffb700 1 -- 172.21.15.190:0/2932426143 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f013c018070 con 0x7f014c0601e0 2024-01-16T17:18:57.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c106dd0 con 0x7f014c0601e0 2024-01-16T17:18:57.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f0141ffb700 1 -- 172.21.15.190:0/2932426143 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f013c013410 con 0x7f014c0601e0 2024-01-16T17:18:57.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f0141ffb700 1 -- 172.21.15.190:0/2932426143 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f013c01c3c0 con 0x7f014c0601e0 2024-01-16T17:18:57.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.740+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014c107260 con 0x7f014c0601e0 2024-01-16T17:18:57.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.741+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014c045ad0 con 0x7f014c0601e0 2024-01-16T17:18:57.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.741+0000 7f0141ffb700 1 -- 172.21.15.190:0/2932426143 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f013c01d420 con 0x7f014c0601e0 2024-01-16T17:18:57.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.741+0000 7f0141ffb700 1 --2- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f013802ff60 0x7f0138032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:18:57.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.742+0000 7f0141ffb700 1 -- 172.21.15.190:0/2932426143 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f013c041390 con 0x7f014c0601e0 2024-01-16T17:18:57.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.742+0000 7f0150f51700 1 --2- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f013802ff60 0x7f0138032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:18:57.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.742+0000 7f0150f51700 1 --2- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f013802ff60 0x7f0138032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f01480096a0 tx=0x7f0148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:18:57.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.746+0000 7f0141ffb700 1 -- 172.21.15.190:0/2932426143 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f013c01d820 con 0x7f014c0601e0 2024-01-16T17:18:57.927 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.949+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f014c1080b0 con 0x7f014c0601e0 2024-01-16T17:18:57.928 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.950+0000 7f0141ffb700 1 -- 172.21.15.190:0/2932426143 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f013c01b370 con 0x7f014c0601e0 2024-01-16T17:18:57.928 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:18:57.928 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:18:57.929 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f013802ff60 msgr2=0x7f0138032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 --2- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f013802ff60 0x7f0138032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f01480096a0 tx=0x7f0148006b40).stop 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 msgr2=0x7f014c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 --2- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 0x7f014c1066e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f013c004dd0 tx=0x7f013c009b30).stop 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 shutdown_connections 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 --2- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f013802ff60 0x7f0138032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 --2- 172.21.15.190:0/2932426143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f014c0601e0 0x7f014c1066e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 >> 172.21.15.190:0/2932426143 conn(0x7f014c0f0b70 msgr2=0x7f014c0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.951+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 shutdown_connections 2024-01-16T17:18:57.930 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:18:57.952+0000 7f01539b6700 1 -- 172.21.15.190:0/2932426143 wait complete. 2024-01-16T17:18:57.931 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:18:57.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:57 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:55.860467+0000 mgr.smithi154.sllivz (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:18:58.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:58 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:18:57.950941+0000 mon.smithi154 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.190:0/2932426143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:18:59.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:18:59.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:18:59.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:18:59 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:57.860834+0000 mgr.smithi154.sllivz (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:00.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.497+0000 7f947531c700 1 -- 172.21.15.190:0/1951646655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 msgr2=0x7f94700f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:00.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.497+0000 7f947531c700 1 --2- 172.21.15.190:0/1951646655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 0x7f94700f4be0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f9464009fb0 tx=0x7f9464009b30).stop 2024-01-16T17:19:00.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.497+0000 7f947531c700 1 -- 172.21.15.190:0/1951646655 shutdown_connections 2024-01-16T17:19:00.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.497+0000 7f947531c700 1 --2- 172.21.15.190:0/1951646655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 0x7f94700f4be0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:00.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.497+0000 7f947531c700 1 -- 172.21.15.190:0/1951646655 >> 172.21.15.190:0/1951646655 conn(0x7f94700f02e0 msgr2=0x7f94700f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:00.476 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.497+0000 7f947531c700 1 -- 172.21.15.190:0/1951646655 shutdown_connections 2024-01-16T17:19:00.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.497+0000 7f947531c700 1 -- 172.21.15.190:0/1951646655 wait complete. 2024-01-16T17:19:00.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.498+0000 7f947531c700 1 Processor -- start 2024-01-16T17:19:00.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.498+0000 7f947531c700 1 -- start start 2024-01-16T17:19:00.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.498+0000 7f947531c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 0x7f9470106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:00.477 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.498+0000 7f947531c700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9470106bf0 con 0x7f94700f47c0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f946effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 0x7f9470106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f946effd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 0x7f9470106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:36752/0 (socket says 172.21.15.190:36752) 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f946effd700 1 -- 172.21.15.190:0/2160541710 learned_addr learned my addr 172.21.15.190:0/2160541710 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f946effd700 1 -- 172.21.15.190:0/2160541710 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9464005040 con 0x7f94700f47c0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f946effd700 1 --2- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 0x7f9470106700 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f9464013510 tx=0x7f946400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f94577fe700 1 -- 172.21.15.190:0/2160541710 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9464009320 con 0x7f94700f47c0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9470106df0 con 0x7f94700f47c0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f94577fe700 1 -- 172.21.15.190:0/2160541710 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9464013f00 con 0x7f94700f47c0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f94577fe700 1 -- 172.21.15.190:0/2160541710 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9464014d90 con 0x7f94700f47c0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.499+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9470107280 con 0x7f94700f47c0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.500+0000 7f94577fe700 1 -- 172.21.15.190:0/2160541710 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9464014420 con 0x7f94700f47c0 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.500+0000 7f94577fe700 1 --2- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f945802ff60 0x7f9458032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.500+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9470045ad0 con 0x7f94700f47c0 2024-01-16T17:19:00.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.500+0000 7f94577fe700 1 -- 172.21.15.190:0/2160541710 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94640410c0 con 0x7f94700f47c0 2024-01-16T17:19:00.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.501+0000 7f946e7fc700 1 --2- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f945802ff60 0x7f9458032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:00.480 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.501+0000 7f946e7fc700 1 --2- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f945802ff60 0x7f9458032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f94600096a0 tx=0x7f9460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:00.483 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.505+0000 7f94577fe700 1 -- 172.21.15.190:0/2160541710 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f946406d2b0 con 0x7f94700f47c0 2024-01-16T17:19:00.692 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.714+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9470108130 con 0x7f94700f47c0 2024-01-16T17:19:00.693 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.715+0000 7f94577fe700 1 -- 172.21.15.190:0/2160541710 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f946401dd50 con 0x7f94700f47c0 2024-01-16T17:19:00.693 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:00.693 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:00.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f945802ff60 msgr2=0x7f9458032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:00.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 --2- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f945802ff60 0x7f9458032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f94600096a0 tx=0x7f9460006b40).stop 2024-01-16T17:19:00.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 msgr2=0x7f9470106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:00.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 --2- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 0x7f9470106700 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f9464013510 tx=0x7f946400b040).stop 2024-01-16T17:19:00.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 shutdown_connections 2024-01-16T17:19:00.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 --2- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f945802ff60 0x7f9458032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:00.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 --2- 172.21.15.190:0/2160541710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94700f47c0 0x7f9470106700 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:00.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 >> 172.21.15.190:0/2160541710 conn(0x7f94700f02e0 msgr2=0x7f94700f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:00.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 shutdown_connections 2024-01-16T17:19:00.695 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:00.716+0000 7f947531c700 1 -- 172.21.15.190:0/2160541710 wait complete. 2024-01-16T17:19:00.695 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:01.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:18:59.861180+0000 mgr.smithi154.sllivz (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:01.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:01 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:00.715500+0000 mon.smithi154 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.190:0/2160541710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:02.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:02.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:03.163 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.183+0000 7f8a1447b700 1 -- 172.21.15.190:0/1818192580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 msgr2=0x7f8a0c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:03.163 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.183+0000 7f8a1447b700 1 --2- 172.21.15.190:0/1818192580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 0x7f8a0c0f4be0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f89fc009fb0 tx=0x7f89fc009b30).stop 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 -- 172.21.15.190:0/1818192580 shutdown_connections 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 --2- 172.21.15.190:0/1818192580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 0x7f8a0c0f4be0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 -- 172.21.15.190:0/1818192580 >> 172.21.15.190:0/1818192580 conn(0x7f8a0c0f02e0 msgr2=0x7f8a0c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 -- 172.21.15.190:0/1818192580 shutdown_connections 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 -- 172.21.15.190:0/1818192580 wait complete. 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 Processor -- start 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 -- start start 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 0x7f8a0c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a1447b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a0c106bf0 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.164 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a12217700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 0x7f8a0c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.184+0000 7f8a12217700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 0x7f8a0c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60066/0 (socket says 172.21.15.190:60066) 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.185+0000 7f8a12217700 1 -- 172.21.15.190:0/1385221910 learned_addr learned my addr 172.21.15.190:0/1385221910 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.185+0000 7f8a12217700 1 -- 172.21.15.190:0/1385221910 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc005040 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.185+0000 7f8a12217700 1 --2- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 0x7f8a0c106700 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f89fc013510 tx=0x7f89fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.185+0000 7f8a02ffd700 1 -- 172.21.15.190:0/1385221910 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89fc009320 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.185+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a0c106df0 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.185+0000 7f8a02ffd700 1 -- 172.21.15.190:0/1385221910 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89fc013f00 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.185+0000 7f8a02ffd700 1 -- 172.21.15.190:0/1385221910 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89fc014d90 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.165 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.185+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a0c107280 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.186+0000 7f8a02ffd700 1 -- 172.21.15.190:0/1385221910 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89fc0095f0 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.186+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a0c045ad0 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.166 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.186+0000 7f8a02ffd700 1 --2- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f89f802ff60 0x7f89f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:03.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.186+0000 7f8a02ffd700 1 -- 172.21.15.190:0/1385221910 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89fc0410c0 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.186+0000 7f8a11a16700 1 --2- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f89f802ff60 0x7f89f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:03.167 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.187+0000 7f8a11a16700 1 --2- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f89f802ff60 0x7f89f8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8a080096a0 tx=0x7f8a08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:03.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.190+0000 7f8a02ffd700 1 -- 172.21.15.190:0/1385221910 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89fc014420 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.373 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.395+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a0c108130 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.374 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.396+0000 7f8a02ffd700 1 -- 172.21.15.190:0/1385221910 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f89fc010230 con 0x7f8a0c0f47c0 2024-01-16T17:19:03.374 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:03.374 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:03.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f89f802ff60 msgr2=0x7f89f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:03.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 --2- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f89f802ff60 0x7f89f8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8a080096a0 tx=0x7f8a08006b40).stop 2024-01-16T17:19:03.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 msgr2=0x7f8a0c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:03.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 --2- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 0x7f8a0c106700 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f89fc013510 tx=0x7f89fc00b040).stop 2024-01-16T17:19:03.376 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 shutdown_connections 2024-01-16T17:19:03.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 --2- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f89f802ff60 0x7f89f8032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:03.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 --2- 172.21.15.190:0/1385221910 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a0c0f47c0 0x7f8a0c106700 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:03.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 >> 172.21.15.190:0/1385221910 conn(0x7f8a0c0f02e0 msgr2=0x7f8a0c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:03.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 shutdown_connections 2024-01-16T17:19:03.377 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:03.398+0000 7f8a1447b700 1 -- 172.21.15.190:0/1385221910 wait complete. 2024-01-16T17:19:03.377 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:03.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:01.861517+0000 mgr.smithi154.sllivz (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:03.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:03 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:03.397347+0000 mon.smithi154 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.190:0/1385221910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:04.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:04.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:05.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.925+0000 7f229043a700 1 -- 172.21.15.190:0/2126122671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 msgr2=0x7f22880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:05.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.925+0000 7f229043a700 1 --2- 172.21.15.190:0/2126122671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 0x7f22880f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f2278009fb0 tx=0x7f2278009b30).stop 2024-01-16T17:19:05.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.925+0000 7f229043a700 1 -- 172.21.15.190:0/2126122671 shutdown_connections 2024-01-16T17:19:05.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.925+0000 7f229043a700 1 --2- 172.21.15.190:0/2126122671 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 0x7f22880f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:05.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.925+0000 7f229043a700 1 -- 172.21.15.190:0/2126122671 >> 172.21.15.190:0/2126122671 conn(0x7f22880f0b70 msgr2=0x7f22880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:05.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.925+0000 7f229043a700 1 -- 172.21.15.190:0/2126122671 shutdown_connections 2024-01-16T17:19:05.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.926+0000 7f229043a700 1 -- 172.21.15.190:0/2126122671 wait complete. 2024-01-16T17:19:05.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.926+0000 7f229043a700 1 Processor -- start 2024-01-16T17:19:05.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.926+0000 7f229043a700 1 -- start start 2024-01-16T17:19:05.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.926+0000 7f229043a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 0x7f2288109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:05.905 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.926+0000 7f229043a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2288060240 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f228e1d6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 0x7f2288109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f228e1d6700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 0x7f2288109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60078/0 (socket says 172.21.15.190:60078) 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f228e1d6700 1 -- 172.21.15.190:0/2820693351 learned_addr learned my addr 172.21.15.190:0/2820693351 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f228e1d6700 1 -- 172.21.15.190:0/2820693351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2278005040 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f228e1d6700 1 --2- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 0x7f2288109400 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f2278014000 tx=0x7f227800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f227effd700 1 -- 172.21.15.190:0/2820693351 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f22780095f0 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f227effd700 1 -- 172.21.15.190:0/2820693351 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2278007e70 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2288060440 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f227effd700 1 -- 172.21.15.190:0/2820693351 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2278014ea0 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.927+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2288109ab0 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.928+0000 7f227effd700 1 -- 172.21.15.190:0/2820693351 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2278025070 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.928+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2288045ad0 con 0x7f22880f5410 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.928+0000 7f227effd700 1 --2- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f227402ff60 0x7f2274032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:05.907 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.928+0000 7f227effd700 1 -- 172.21.15.190:0/2820693351 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f22780409b0 con 0x7f22880f5410 2024-01-16T17:19:05.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.928+0000 7f228d9d5700 1 --2- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f227402ff60 0x7f2274032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:05.908 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.929+0000 7f228d9d5700 1 --2- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f227402ff60 0x7f2274032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f22840096a0 tx=0x7f2284006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:05.911 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:05.933+0000 7f227effd700 1 -- 172.21.15.190:0/2820693351 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f227800f4e0 con 0x7f22880f5410 2024-01-16T17:19:05.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:05 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:03.861860+0000 mgr.smithi154.sllivz (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:06.118 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.139+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f22880fae60 con 0x7f22880f5410 2024-01-16T17:19:06.119 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.140+0000 7f227effd700 1 -- 172.21.15.190:0/2820693351 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f227800f300 con 0x7f22880f5410 2024-01-16T17:19:06.119 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:06.119 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f227402ff60 msgr2=0x7f2274032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 --2- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f227402ff60 0x7f2274032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f22840096a0 tx=0x7f2284006b40).stop 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 msgr2=0x7f2288109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 --2- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 0x7f2288109400 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f2278014000 tx=0x7f227800b040).stop 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 shutdown_connections 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 --2- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f227402ff60 0x7f2274032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 --2- 172.21.15.190:0/2820693351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f22880f5410 0x7f2288109400 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 >> 172.21.15.190:0/2820693351 conn(0x7f22880f0b70 msgr2=0x7f22880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 shutdown_connections 2024-01-16T17:19:06.121 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:06.142+0000 7f229043a700 1 -- 172.21.15.190:0/2820693351 wait complete. 2024-01-16T17:19:06.122 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:06.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:06 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:06.142112+0000 mon.smithi154 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.190:0/2820693351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:07.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:07.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:07.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:07 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:05.862205+0000 mgr.smithi154.sllivz (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:08.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.707+0000 7f550b957700 1 -- 172.21.15.190:0/2428281285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 msgr2=0x7f55040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:08.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.707+0000 7f550b957700 1 --2- 172.21.15.190:0/2428281285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 0x7f55040f6930 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f54f4009fb0 tx=0x7f54f4009b30).stop 2024-01-16T17:19:08.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 -- 172.21.15.190:0/2428281285 shutdown_connections 2024-01-16T17:19:08.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 --2- 172.21.15.190:0/2428281285 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 0x7f55040f6930 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:08.689 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 -- 172.21.15.190:0/2428281285 >> 172.21.15.190:0/2428281285 conn(0x7f55040f1c70 msgr2=0x7f55040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 -- 172.21.15.190:0/2428281285 shutdown_connections 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 -- 172.21.15.190:0/2428281285 wait complete. 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 Processor -- start 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 -- start start 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 0x7f5504106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f550b957700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5504106c20 con 0x7f55040f6510 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f55096f3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 0x7f5504106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f55096f3700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 0x7f5504106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60104/0 (socket says 172.21.15.190:60104) 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.708+0000 7f55096f3700 1 -- 172.21.15.190:0/1162754982 learned_addr learned my addr 172.21.15.190:0/1162754982 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.709+0000 7f55096f3700 1 -- 172.21.15.190:0/1162754982 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f4005040 con 0x7f55040f6510 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.709+0000 7f55096f3700 1 --2- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 0x7f5504106730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f54f4013510 tx=0x7f54f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.709+0000 7f54f9ffb700 1 -- 172.21.15.190:0/1162754982 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f4009320 con 0x7f55040f6510 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.709+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5504106e20 con 0x7f55040f6510 2024-01-16T17:19:08.690 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.709+0000 7f54f9ffb700 1 -- 172.21.15.190:0/1162754982 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54f4013f00 con 0x7f55040f6510 2024-01-16T17:19:08.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.709+0000 7f54f9ffb700 1 -- 172.21.15.190:0/1162754982 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f4014d90 con 0x7f55040f6510 2024-01-16T17:19:08.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.709+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55041072b0 con 0x7f55040f6510 2024-01-16T17:19:08.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.710+0000 7f54f9ffb700 1 -- 172.21.15.190:0/1162754982 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54f4014420 con 0x7f55040f6510 2024-01-16T17:19:08.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.710+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5504045ad0 con 0x7f55040f6510 2024-01-16T17:19:08.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.710+0000 7f54f9ffb700 1 --2- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f54f002ff60 0x7f54f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:08.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.710+0000 7f54f9ffb700 1 -- 172.21.15.190:0/1162754982 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54f4041110 con 0x7f55040f6510 2024-01-16T17:19:08.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.710+0000 7f5508ef2700 1 --2- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f54f002ff60 0x7f54f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:08.691 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.711+0000 7f5508ef2700 1 --2- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f54f002ff60 0x7f54f0032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f55000096a0 tx=0x7f5500006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:08.694 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.715+0000 7f54f9ffb700 1 -- 172.21.15.190:0/1162754982 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54f40095f0 con 0x7f55040f6510 2024-01-16T17:19:08.900 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.921+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55040fbf60 con 0x7f55040f6510 2024-01-16T17:19:08.901 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.922+0000 7f54f9ffb700 1 -- 172.21.15.190:0/1162754982 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f54f401db70 con 0x7f55040f6510 2024-01-16T17:19:08.901 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:08.901 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f54f002ff60 msgr2=0x7f54f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 --2- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f54f002ff60 0x7f54f0032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f55000096a0 tx=0x7f5500006b40).stop 2024-01-16T17:19:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 msgr2=0x7f5504106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:08.903 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 --2- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 0x7f5504106730 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f54f4013510 tx=0x7f54f400b040).stop 2024-01-16T17:19:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 shutdown_connections 2024-01-16T17:19:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 --2- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f54f002ff60 0x7f54f0032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 --2- 172.21.15.190:0/1162754982 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55040f6510 0x7f5504106730 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 >> 172.21.15.190:0/1162754982 conn(0x7f55040f1c70 msgr2=0x7f55040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 shutdown_connections 2024-01-16T17:19:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:08.924+0000 7f550b957700 1 -- 172.21.15.190:0/1162754982 wait complete. 2024-01-16T17:19:08.904 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:09.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:07.862517+0000 mgr.smithi154.sllivz (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:09.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:09 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:08.923958+0000 mon.smithi154 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.190:0/1162754982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:10.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:10.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:11.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.534+0000 7fec96986700 1 -- 172.21.15.190:0/3003306841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 msgr2=0x7fec900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:11.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.534+0000 7fec96986700 1 --2- 172.21.15.190:0/3003306841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 0x7fec900f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fec80009fb0 tx=0x7fec80009b30).stop 2024-01-16T17:19:11.517 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.534+0000 7fec96986700 1 -- 172.21.15.190:0/3003306841 shutdown_connections 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.534+0000 7fec96986700 1 --2- 172.21.15.190:0/3003306841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 0x7fec900f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.534+0000 7fec96986700 1 -- 172.21.15.190:0/3003306841 >> 172.21.15.190:0/3003306841 conn(0x7fec900f1c70 msgr2=0x7fec900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.534+0000 7fec96986700 1 -- 172.21.15.190:0/3003306841 shutdown_connections 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.534+0000 7fec96986700 1 -- 172.21.15.190:0/3003306841 wait complete. 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.535+0000 7fec96986700 1 Processor -- start 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.535+0000 7fec96986700 1 -- start start 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.535+0000 7fec96986700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 0x7fec901066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.535+0000 7fec96986700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90106bc0 con 0x7fec900f6510 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.535+0000 7fec8ffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 0x7fec901066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.535+0000 7fec8ffff700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 0x7fec901066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60130/0 (socket says 172.21.15.190:60130) 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.535+0000 7fec8ffff700 1 -- 172.21.15.190:0/1574409993 learned_addr learned my addr 172.21.15.190:0/1574409993 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.536+0000 7fec8ffff700 1 -- 172.21.15.190:0/1574409993 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80005040 con 0x7fec900f6510 2024-01-16T17:19:11.518 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.536+0000 7fec8ffff700 1 --2- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 0x7fec901066d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fec80013510 tx=0x7fec8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.536+0000 7fec8cff9700 1 -- 172.21.15.190:0/1574409993 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec80009320 con 0x7fec900f6510 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.536+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90106dc0 con 0x7fec900f6510 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.536+0000 7fec8cff9700 1 -- 172.21.15.190:0/1574409993 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec80013f00 con 0x7fec900f6510 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.536+0000 7fec8cff9700 1 -- 172.21.15.190:0/1574409993 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec80014d90 con 0x7fec900f6510 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.536+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90107250 con 0x7fec900f6510 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.537+0000 7fec8cff9700 1 -- 172.21.15.190:0/1574409993 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec800263a0 con 0x7fec900f6510 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.537+0000 7fec8cff9700 1 --2- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.537+0000 7fec8cff9700 1 -- 172.21.15.190:0/1574409993 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec80041090 con 0x7fec900f6510 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.537+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec90045ad0 con 0x7fec900f6510 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.537+0000 7fec8f7fe700 1 --2- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:11.519 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.537+0000 7fec8f7fe700 1 --2- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fec7c02ff60 0x7fec7c032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fec780096a0 tx=0x7fec78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:11.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.542+0000 7fec8cff9700 1 -- 172.21.15.190:0/1574409993 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec80012920 con 0x7fec900f6510 2024-01-16T17:19:11.726 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.747+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec90107f70 con 0x7fec900f6510 2024-01-16T17:19:11.727 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.748+0000 7fec8cff9700 1 -- 172.21.15.190:0/1574409993 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec8002b020 con 0x7fec900f6510 2024-01-16T17:19:11.727 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:11.727 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fec7c02ff60 msgr2=0x7fec7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 --2- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fec7c02ff60 0x7fec7c032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fec780096a0 tx=0x7fec78006b40).stop 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 msgr2=0x7fec901066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 --2- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 0x7fec901066d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fec80013510 tx=0x7fec8000b040).stop 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 shutdown_connections 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 --2- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fec7c02ff60 0x7fec7c032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 --2- 172.21.15.190:0/1574409993 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec900f6510 0x7fec901066d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 >> 172.21.15.190:0/1574409993 conn(0x7fec900f1c70 msgr2=0x7fec900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 shutdown_connections 2024-01-16T17:19:11.729 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:11.750+0000 7fec96986700 1 -- 172.21.15.190:0/1574409993 wait complete. 2024-01-16T17:19:11.730 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:11.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:11 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:09.862931+0000 mgr.smithi154.sllivz (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:12.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:12 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:11.750178+0000 mon.smithi154 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.190:0/1574409993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:13.063 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:13.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:13.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:13 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:11.863278+0000 mgr.smithi154.sllivz (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:14.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.192+0000 7f9043999700 1 -- 172.21.15.190:0/2826677186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 msgr2=0x7f903c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.192+0000 7f9043999700 1 --2- 172.21.15.190:0/2826677186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 0x7f903c0f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f902c009fb0 tx=0x7f902c009b30).stop 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.192+0000 7f9043999700 1 -- 172.21.15.190:0/2826677186 shutdown_connections 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.192+0000 7f9043999700 1 --2- 172.21.15.190:0/2826677186 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 0x7f903c0f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.192+0000 7f9043999700 1 -- 172.21.15.190:0/2826677186 >> 172.21.15.190:0/2826677186 conn(0x7f903c0f1c70 msgr2=0x7f903c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.192+0000 7f9043999700 1 -- 172.21.15.190:0/2826677186 shutdown_connections 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.192+0000 7f9043999700 1 -- 172.21.15.190:0/2826677186 wait complete. 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.193+0000 7f9043999700 1 Processor -- start 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.193+0000 7f9043999700 1 -- start start 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.193+0000 7f9043999700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 0x7f903c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.193+0000 7f9043999700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903c069e20 con 0x7f903c0f6510 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.193+0000 7f9041735700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 0x7f903c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:14.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9041735700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 0x7f903c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:39918/0 (socket says 172.21.15.190:39918) 2024-01-16T17:19:14.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9041735700 1 -- 172.21.15.190:0/1247225874 learned_addr learned my addr 172.21.15.190:0/1247225874 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:14.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9041735700 1 -- 172.21.15.190:0/1247225874 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c005040 con 0x7f903c0f6510 2024-01-16T17:19:14.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9041735700 1 --2- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 0x7f903c06b9a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f902c013510 tx=0x7f902c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:14.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9031ffb700 1 -- 172.21.15.190:0/1247225874 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f902c009320 con 0x7f903c0f6510 2024-01-16T17:19:14.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903c069fc0 con 0x7f903c0f6510 2024-01-16T17:19:14.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9031ffb700 1 -- 172.21.15.190:0/1247225874 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f902c013f00 con 0x7f903c0f6510 2024-01-16T17:19:14.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9031ffb700 1 -- 172.21.15.190:0/1247225874 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f902c014d90 con 0x7f903c0f6510 2024-01-16T17:19:14.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.194+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903c06a450 con 0x7f903c0f6510 2024-01-16T17:19:14.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.195+0000 7f9031ffb700 1 -- 172.21.15.190:0/1247225874 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f902c0095f0 con 0x7f903c0f6510 2024-01-16T17:19:14.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.195+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903c059100 con 0x7f903c0f6510 2024-01-16T17:19:14.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.195+0000 7f9031ffb700 1 --2- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f902802ff60 0x7f9028032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:14.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.195+0000 7f9031ffb700 1 -- 172.21.15.190:0/1247225874 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f902c041000 con 0x7f903c0f6510 2024-01-16T17:19:14.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.195+0000 7f9040f34700 1 --2- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f902802ff60 0x7f9028032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:14.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.196+0000 7f9040f34700 1 --2- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f902802ff60 0x7f9028032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f90380096a0 tx=0x7f9038006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:14.178 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.200+0000 7f9031ffb700 1 -- 172.21.15.190:0/1247225874 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f902c014420 con 0x7f903c0f6510 2024-01-16T17:19:14.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.407+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f903c0fbf60 con 0x7f903c0f6510 2024-01-16T17:19:14.386 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.407+0000 7f9031ffb700 1 -- 172.21.15.190:0/1247225874 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f902c01d7f0 con 0x7f903c0f6510 2024-01-16T17:19:14.386 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:14.386 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:14.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f902802ff60 msgr2=0x7f9028032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:14.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 --2- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f902802ff60 0x7f9028032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f90380096a0 tx=0x7f9038006b40).stop 2024-01-16T17:19:14.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 msgr2=0x7f903c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:14.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 --2- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 0x7f903c06b9a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f902c013510 tx=0x7f902c00b040).stop 2024-01-16T17:19:14.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 shutdown_connections 2024-01-16T17:19:14.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 --2- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f902802ff60 0x7f9028032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:14.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 --2- 172.21.15.190:0/1247225874 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f903c0f6510 0x7f903c06b9a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:14.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 >> 172.21.15.190:0/1247225874 conn(0x7f903c0f1c70 msgr2=0x7f903c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:14.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 shutdown_connections 2024-01-16T17:19:14.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:14.409+0000 7f9043999700 1 -- 172.21.15.190:0/1247225874 wait complete. 2024-01-16T17:19:14.389 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:14.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:14 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:14.409584+0000 mon.smithi154 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.190:0/1247225874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:15.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:15.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:15.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:15 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:13.863634+0000 mgr.smithi154.sllivz (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.876+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/2645915764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 msgr2=0x7f2d980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.876+0000 7f2d9ecb8700 1 --2- 172.21.15.190:0/2645915764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 0x7f2d980f6930 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2d8c009fb0 tx=0x7f2d8c009b30).stop 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.876+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/2645915764 shutdown_connections 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.876+0000 7f2d9ecb8700 1 --2- 172.21.15.190:0/2645915764 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 0x7f2d980f6930 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.876+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/2645915764 >> 172.21.15.190:0/2645915764 conn(0x7f2d980f1c70 msgr2=0x7f2d980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.876+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/2645915764 shutdown_connections 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.876+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/2645915764 wait complete. 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.877+0000 7f2d9ecb8700 1 Processor -- start 2024-01-16T17:19:16.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.877+0000 7f2d9ecb8700 1 -- start start 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.877+0000 7f2d9ecb8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 0x7f2d98106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.877+0000 7f2d9ecb8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98106c20 con 0x7f2d980f6510 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.877+0000 7f2d9ca54700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 0x7f2d98106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.877+0000 7f2d9ca54700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 0x7f2d98106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:39934/0 (socket says 172.21.15.190:39934) 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.877+0000 7f2d9ca54700 1 -- 172.21.15.190:0/147962159 learned_addr learned my addr 172.21.15.190:0/147962159 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.878+0000 7f2d9ca54700 1 -- 172.21.15.190:0/147962159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c005040 con 0x7f2d980f6510 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.878+0000 7f2d9ca54700 1 --2- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 0x7f2d98106730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f2d8c013510 tx=0x7f2d8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.878+0000 7f2d957fa700 1 -- 172.21.15.190:0/147962159 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8c009320 con 0x7f2d980f6510 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.878+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d98106e20 con 0x7f2d980f6510 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.878+0000 7f2d957fa700 1 -- 172.21.15.190:0/147962159 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d8c013f00 con 0x7f2d980f6510 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.878+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d981072b0 con 0x7f2d980f6510 2024-01-16T17:19:16.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.878+0000 7f2d957fa700 1 -- 172.21.15.190:0/147962159 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d8c014d90 con 0x7f2d980f6510 2024-01-16T17:19:16.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.879+0000 7f2d957fa700 1 -- 172.21.15.190:0/147962159 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d8c014420 con 0x7f2d980f6510 2024-01-16T17:19:16.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.879+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d98045ad0 con 0x7f2d980f6510 2024-01-16T17:19:16.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.879+0000 7f2d957fa700 1 --2- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2d8802ff60 0x7f2d88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:16.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.879+0000 7f2d957fa700 1 -- 172.21.15.190:0/147962159 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d8c041020 con 0x7f2d980f6510 2024-01-16T17:19:16.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.879+0000 7f2d97fff700 1 --2- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2d8802ff60 0x7f2d88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:16.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.880+0000 7f2d97fff700 1 --2- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2d8802ff60 0x7f2d88032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2d840096a0 tx=0x7f2d84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:16.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:16.883+0000 7f2d957fa700 1 -- 172.21.15.190:0/147962159 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d8c0095f0 con 0x7f2d980f6510 2024-01-16T17:19:17.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.093+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d980fbf60 con 0x7f2d980f6510 2024-01-16T17:19:17.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.094+0000 7f2d957fa700 1 -- 172.21.15.190:0/147962159 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2d8c01db70 con 0x7f2d980f6510 2024-01-16T17:19:17.072 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:17.072 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:17.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2d8802ff60 msgr2=0x7f2d88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 --2- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2d8802ff60 0x7f2d88032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2d840096a0 tx=0x7f2d84006b40).stop 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 msgr2=0x7f2d98106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 --2- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 0x7f2d98106730 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f2d8c013510 tx=0x7f2d8c00b040).stop 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 shutdown_connections 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 --2- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f2d8802ff60 0x7f2d88032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 --2- 172.21.15.190:0/147962159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d980f6510 0x7f2d98106730 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 >> 172.21.15.190:0/147962159 conn(0x7f2d980f1c70 msgr2=0x7f2d980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 shutdown_connections 2024-01-16T17:19:17.074 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:17.096+0000 7f2d9ecb8700 1 -- 172.21.15.190:0/147962159 wait complete. 2024-01-16T17:19:17.075 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:17.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:15.863941+0000 mgr.smithi154.sllivz (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:17.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:17 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:17.095303+0000 mon.smithi154 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.190:0/147962159' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:18.434 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:18.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.576+0000 7fcb2c294700 1 -- 172.21.15.190:0/2016463506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 msgr2=0x7fcb240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.576+0000 7fcb2c294700 1 --2- 172.21.15.190:0/2016463506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 0x7fcb240f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fcb14009fb0 tx=0x7fcb14009b30).stop 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.576+0000 7fcb2c294700 1 -- 172.21.15.190:0/2016463506 shutdown_connections 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.576+0000 7fcb2c294700 1 --2- 172.21.15.190:0/2016463506 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 0x7fcb240f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.576+0000 7fcb2c294700 1 -- 172.21.15.190:0/2016463506 >> 172.21.15.190:0/2016463506 conn(0x7fcb240f1c70 msgr2=0x7fcb240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.576+0000 7fcb2c294700 1 -- 172.21.15.190:0/2016463506 shutdown_connections 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.576+0000 7fcb2c294700 1 -- 172.21.15.190:0/2016463506 wait complete. 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.577+0000 7fcb2c294700 1 Processor -- start 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.577+0000 7fcb2c294700 1 -- start start 2024-01-16T17:19:19.555 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.577+0000 7fcb2c294700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 0x7fcb2406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:19.556 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.577+0000 7fcb2c294700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb24069e20 con 0x7fcb240f6510 2024-01-16T17:19:19.556 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.577+0000 7fcb2a030700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 0x7fcb2406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:19.556 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.577+0000 7fcb2a030700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 0x7fcb2406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:39958/0 (socket says 172.21.15.190:39958) 2024-01-16T17:19:19.556 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.577+0000 7fcb2a030700 1 -- 172.21.15.190:0/3461796007 learned_addr learned my addr 172.21.15.190:0/3461796007 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:19.557 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.578+0000 7fcb2a030700 1 -- 172.21.15.190:0/3461796007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb14005040 con 0x7fcb240f6510 2024-01-16T17:19:19.557 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.578+0000 7fcb2a030700 1 --2- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 0x7fcb2406b9a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fcb14014000 tx=0x7fcb1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:19.557 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.578+0000 7fcb1affd700 1 -- 172.21.15.190:0/3461796007 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb140095f0 con 0x7fcb240f6510 2024-01-16T17:19:19.557 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.578+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb24069fc0 con 0x7fcb240f6510 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.578+0000 7fcb1affd700 1 -- 172.21.15.190:0/3461796007 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb14007e70 con 0x7fcb240f6510 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.578+0000 7fcb1affd700 1 -- 172.21.15.190:0/3461796007 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb14014ea0 con 0x7fcb240f6510 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.578+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb2406a450 con 0x7fcb240f6510 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.579+0000 7fcb1affd700 1 -- 172.21.15.190:0/3461796007 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb14025070 con 0x7fcb240f6510 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.579+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb24059100 con 0x7fcb240f6510 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.579+0000 7fcb1affd700 1 --2- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb1002ffb0 0x7fcb10032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.579+0000 7fcb1affd700 1 -- 172.21.15.190:0/3461796007 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb140409f0 con 0x7fcb240f6510 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.579+0000 7fcb2982f700 1 --2- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb1002ffb0 0x7fcb10032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:19.558 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.579+0000 7fcb2982f700 1 --2- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb1002ffb0 0x7fcb10032470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fcb200096a0 tx=0x7fcb20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:19.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.584+0000 7fcb1affd700 1 -- 172.21.15.190:0/3461796007 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb140079c0 con 0x7fcb240f6510 2024-01-16T17:19:19.769 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.791+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb240fbf60 con 0x7fcb240f6510 2024-01-16T17:19:19.769 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.791+0000 7fcb1affd700 1 -- 172.21.15.190:0/3461796007 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb1400f300 con 0x7fcb240f6510 2024-01-16T17:19:19.769 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:19.769 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb1002ffb0 msgr2=0x7fcb10032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 --2- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb1002ffb0 0x7fcb10032470 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fcb200096a0 tx=0x7fcb20006b40).stop 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 msgr2=0x7fcb2406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 --2- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 0x7fcb2406b9a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fcb14014000 tx=0x7fcb1400b040).stop 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 shutdown_connections 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 --2- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fcb1002ffb0 0x7fcb10032470 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 --2- 172.21.15.190:0/3461796007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240f6510 0x7fcb2406b9a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 >> 172.21.15.190:0/3461796007 conn(0x7fcb240f1c70 msgr2=0x7fcb240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 shutdown_connections 2024-01-16T17:19:19.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:19.793+0000 7fcb2c294700 1 -- 172.21.15.190:0/3461796007 wait complete. 2024-01-16T17:19:19.772 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:19.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:19 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:17.864213+0000 mgr.smithi154.sllivz (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:20.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:20 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:19.792655+0000 mon.smithi154 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.190:0/3461796007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:21.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:21.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:21.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:21 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:19.864551+0000 mgr.smithi154.sllivz (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:22.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 -- 172.21.15.190:0/120186529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 msgr2=0x7fac1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:22.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 --2- 172.21.15.190:0/120186529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 0x7fac1c0f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fac0c009fb0 tx=0x7fac0c009b30).stop 2024-01-16T17:19:22.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 -- 172.21.15.190:0/120186529 shutdown_connections 2024-01-16T17:19:22.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 --2- 172.21.15.190:0/120186529 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 0x7fac1c0f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:22.298 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 -- 172.21.15.190:0/120186529 >> 172.21.15.190:0/120186529 conn(0x7fac1c0f0b70 msgr2=0x7fac1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 -- 172.21.15.190:0/120186529 shutdown_connections 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 -- 172.21.15.190:0/120186529 wait complete. 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 Processor -- start 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 -- start start 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 0x7fac1c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.319+0000 7fac243a8700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac1c060240 con 0x7fac1c0f5410 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac22144700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 0x7fac1c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac22144700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 0x7fac1c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:34066/0 (socket says 172.21.15.190:34066) 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac22144700 1 -- 172.21.15.190:0/79171235 learned_addr learned my addr 172.21.15.190:0/79171235 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac22144700 1 -- 172.21.15.190:0/79171235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c005040 con 0x7fac1c0f5410 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac22144700 1 --2- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 0x7fac1c109400 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fac0c013510 tx=0x7fac0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac12ffd700 1 -- 172.21.15.190:0/79171235 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac0c009320 con 0x7fac1c0f5410 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1c060440 con 0x7fac1c0f5410 2024-01-16T17:19:22.299 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac12ffd700 1 -- 172.21.15.190:0/79171235 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac0c013f00 con 0x7fac1c0f5410 2024-01-16T17:19:22.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac12ffd700 1 -- 172.21.15.190:0/79171235 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac0c014d90 con 0x7fac1c0f5410 2024-01-16T17:19:22.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.320+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1c109ab0 con 0x7fac1c0f5410 2024-01-16T17:19:22.300 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.321+0000 7fac12ffd700 1 -- 172.21.15.190:0/79171235 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac0c0095f0 con 0x7fac1c0f5410 2024-01-16T17:19:22.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.321+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1c045ad0 con 0x7fac1c0f5410 2024-01-16T17:19:22.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.321+0000 7fac12ffd700 1 --2- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac0802ff60 0x7fac08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:22.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.321+0000 7fac12ffd700 1 -- 172.21.15.190:0/79171235 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac0c041000 con 0x7fac1c0f5410 2024-01-16T17:19:22.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.322+0000 7fac21943700 1 --2- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac0802ff60 0x7fac08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:22.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.322+0000 7fac21943700 1 --2- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac0802ff60 0x7fac08032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fac180096a0 tx=0x7fac18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:22.303 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.326+0000 7fac12ffd700 1 -- 172.21.15.190:0/79171235 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac0c014420 con 0x7fac1c0f5410 2024-01-16T17:19:22.510 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.532+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac1c0fae60 con 0x7fac1c0f5410 2024-01-16T17:19:22.510 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.533+0000 7fac12ffd700 1 -- 172.21.15.190:0/79171235 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fac0c01d7f0 con 0x7fac1c0f5410 2024-01-16T17:19:22.511 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:22.511 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:22.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac0802ff60 msgr2=0x7fac08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:22.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 --2- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac0802ff60 0x7fac08032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fac180096a0 tx=0x7fac18006b40).stop 2024-01-16T17:19:22.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 msgr2=0x7fac1c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:22.512 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 --2- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 0x7fac1c109400 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fac0c013510 tx=0x7fac0c00b040).stop 2024-01-16T17:19:22.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 shutdown_connections 2024-01-16T17:19:22.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 --2- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fac0802ff60 0x7fac08032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:22.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 --2- 172.21.15.190:0/79171235 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0f5410 0x7fac1c109400 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:22.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 >> 172.21.15.190:0/79171235 conn(0x7fac1c0f0b70 msgr2=0x7fac1c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:22.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 shutdown_connections 2024-01-16T17:19:22.513 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:22.534+0000 7fac243a8700 1 -- 172.21.15.190:0/79171235 wait complete. 2024-01-16T17:19:22.513 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:22.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:22 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:22.533883+0000 mon.smithi154 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.190:0/79171235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:23.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:23.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:23.955 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:23 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:21.864878+0000 mgr.smithi154.sllivz (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:24.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.006+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2163814156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 msgr2=0x7f0dd40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:24.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.006+0000 7f0dd93f4700 1 --2- 172.21.15.190:0/2163814156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 0x7f0dd40f4be0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f0dc4009fb0 tx=0x7f0dc4009b30).stop 2024-01-16T17:19:24.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2163814156 shutdown_connections 2024-01-16T17:19:24.986 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 --2- 172.21.15.190:0/2163814156 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 0x7f0dd40f4be0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2163814156 >> 172.21.15.190:0/2163814156 conn(0x7f0dd40f02e0 msgr2=0x7f0dd40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2163814156 shutdown_connections 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2163814156 wait complete. 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 Processor -- start 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 -- start start 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 0x7f0dd4106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.007+0000 7f0dd93f4700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4106bf0 con 0x7f0dd40f47c0 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dd2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 0x7f0dd4106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dd2ffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 0x7f0dd4106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:34076/0 (socket says 172.21.15.190:34076) 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dd2ffd700 1 -- 172.21.15.190:0/2679496801 learned_addr learned my addr 172.21.15.190:0/2679496801 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dd2ffd700 1 -- 172.21.15.190:0/2679496801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4005040 con 0x7f0dd40f47c0 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dd2ffd700 1 --2- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 0x7f0dd4106700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f0dc4013510 tx=0x7f0dc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dbb7fe700 1 -- 172.21.15.190:0/2679496801 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dc4009320 con 0x7f0dd40f47c0 2024-01-16T17:19:24.987 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd4106df0 con 0x7f0dd40f47c0 2024-01-16T17:19:24.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dbb7fe700 1 -- 172.21.15.190:0/2679496801 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dc4013f00 con 0x7f0dd40f47c0 2024-01-16T17:19:24.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dbb7fe700 1 -- 172.21.15.190:0/2679496801 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dc4014d90 con 0x7f0dd40f47c0 2024-01-16T17:19:24.988 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.008+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd4107280 con 0x7f0dd40f47c0 2024-01-16T17:19:24.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.009+0000 7f0dbb7fe700 1 -- 172.21.15.190:0/2679496801 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0dc4014420 con 0x7f0dd40f47c0 2024-01-16T17:19:24.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.009+0000 7f0dbb7fe700 1 --2- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0dbc02ff60 0x7f0dbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:24.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.009+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd4045ad0 con 0x7f0dd40f47c0 2024-01-16T17:19:24.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.009+0000 7f0dbb7fe700 1 -- 172.21.15.190:0/2679496801 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0dc4041070 con 0x7f0dd40f47c0 2024-01-16T17:19:24.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.010+0000 7f0dd27fc700 1 --2- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0dbc02ff60 0x7f0dbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:24.989 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.010+0000 7f0dd27fc700 1 --2- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0dbc02ff60 0x7f0dbc032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0dc80096a0 tx=0x7f0dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:24.992 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.014+0000 7f0dbb7fe700 1 -- 172.21.15.190:0/2679496801 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0dc40095f0 con 0x7f0dd40f47c0 2024-01-16T17:19:25.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.220+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0dd4108130 con 0x7f0dd40f47c0 2024-01-16T17:19:25.198 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.220+0000 7f0dbb7fe700 1 -- 172.21.15.190:0/2679496801 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0dc40036f0 con 0x7f0dd40f47c0 2024-01-16T17:19:25.198 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:25.199 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:25.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0dbc02ff60 msgr2=0x7f0dbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:25.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 --2- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0dbc02ff60 0x7f0dbc032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0dc80096a0 tx=0x7f0dc8006b40).stop 2024-01-16T17:19:25.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 msgr2=0x7f0dd4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:25.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 --2- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 0x7f0dd4106700 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f0dc4013510 tx=0x7f0dc400b040).stop 2024-01-16T17:19:25.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 shutdown_connections 2024-01-16T17:19:25.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 --2- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f0dbc02ff60 0x7f0dbc032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:25.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 --2- 172.21.15.190:0/2679496801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0dd40f47c0 0x7f0dd4106700 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:25.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 >> 172.21.15.190:0/2679496801 conn(0x7f0dd40f02e0 msgr2=0x7f0dd40f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:25.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 shutdown_connections 2024-01-16T17:19:25.201 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:25.222+0000 7f0dd93f4700 1 -- 172.21.15.190:0/2679496801 wait complete. 2024-01-16T17:19:25.201 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:25.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:23.865228+0000 mgr.smithi154.sllivz (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:25.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:25 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:25.221601+0000 mon.smithi154 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.190:0/2679496801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:26.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:26.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:27.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.742+0000 7f071a033700 1 -- 172.21.15.190:0/320191720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 msgr2=0x7f07140f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:27.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.742+0000 7f071a033700 1 --2- 172.21.15.190:0/320191720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 0x7f07140f4b60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f0708009fb0 tx=0x7f0708009b30).stop 2024-01-16T17:19:27.720 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.742+0000 7f071a033700 1 -- 172.21.15.190:0/320191720 shutdown_connections 2024-01-16T17:19:27.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.742+0000 7f071a033700 1 --2- 172.21.15.190:0/320191720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 0x7f07140f4b60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:27.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.742+0000 7f071a033700 1 -- 172.21.15.190:0/320191720 >> 172.21.15.190:0/320191720 conn(0x7f07140f0260 msgr2=0x7f07140f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:27.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.742+0000 7f071a033700 1 -- 172.21.15.190:0/320191720 shutdown_connections 2024-01-16T17:19:27.721 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.742+0000 7f071a033700 1 -- 172.21.15.190:0/320191720 wait complete. 2024-01-16T17:19:27.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.743+0000 7f071a033700 1 Processor -- start 2024-01-16T17:19:27.722 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.743+0000 7f071a033700 1 -- start start 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.744+0000 7f071a033700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 0x7f07141066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.744+0000 7f071a033700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0714106bc0 con 0x7f07140f4740 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.744+0000 7f07137fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 0x7f07141066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.744+0000 7f07137fe700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 0x7f07141066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:34104/0 (socket says 172.21.15.190:34104) 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.744+0000 7f07137fe700 1 -- 172.21.15.190:0/4293198944 learned_addr learned my addr 172.21.15.190:0/4293198944 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.744+0000 7f07137fe700 1 -- 172.21.15.190:0/4293198944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0708005040 con 0x7f07140f4740 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.744+0000 7f07137fe700 1 --2- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 0x7f07141066d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0708014000 tx=0x7f070800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.744+0000 7f06fbfff700 1 -- 172.21.15.190:0/4293198944 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07080095f0 con 0x7f07140f4740 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.745+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0714106dc0 con 0x7f07140f4740 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.745+0000 7f06fbfff700 1 -- 172.21.15.190:0/4293198944 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0708007e70 con 0x7f07140f4740 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.745+0000 7f06fbfff700 1 -- 172.21.15.190:0/4293198944 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0708014ea0 con 0x7f07140f4740 2024-01-16T17:19:27.723 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.745+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0714107250 con 0x7f07140f4740 2024-01-16T17:19:27.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.745+0000 7f06fbfff700 1 -- 172.21.15.190:0/4293198944 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0708025070 con 0x7f07140f4740 2024-01-16T17:19:27.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.745+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0714045ad0 con 0x7f07140f4740 2024-01-16T17:19:27.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.745+0000 7f06fbfff700 1 --2- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f06fc02ff60 0x7f06fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:27.724 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.746+0000 7f06fbfff700 1 -- 172.21.15.190:0/4293198944 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0708040a30 con 0x7f07140f4740 2024-01-16T17:19:27.725 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.746+0000 7f0712ffd700 1 --2- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f06fc02ff60 0x7f06fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:27.725 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.746+0000 7f0712ffd700 1 --2- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f06fc02ff60 0x7f06fc032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f07040096a0 tx=0x7f0704006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:27.727 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.750+0000 7f06fbfff700 1 -- 172.21.15.190:0/4293198944 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f070800f4e0 con 0x7f07140f4740 2024-01-16T17:19:27.935 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.957+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0714108100 con 0x7f07140f4740 2024-01-16T17:19:27.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.958+0000 7f06fbfff700 1 -- 172.21.15.190:0/4293198944 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f070800f300 con 0x7f07140f4740 2024-01-16T17:19:27.936 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:27.936 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f06fc02ff60 msgr2=0x7f06fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 --2- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f06fc02ff60 0x7f06fc032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f07040096a0 tx=0x7f0704006b40).stop 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 msgr2=0x7f07141066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 --2- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 0x7f07141066d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0708014000 tx=0x7f070800b040).stop 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 shutdown_connections 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 --2- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f06fc02ff60 0x7f06fc032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 --2- 172.21.15.190:0/4293198944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07140f4740 0x7f07141066d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 >> 172.21.15.190:0/4293198944 conn(0x7f07140f0260 msgr2=0x7f07140f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 shutdown_connections 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:27.960+0000 7f071a033700 1 -- 172.21.15.190:0/4293198944 wait complete. 2024-01-16T17:19:27.938 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:27.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:27 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:25.865617+0000 mgr.smithi154.sllivz (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:28.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:28 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:27.959306+0000 mon.smithi154 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.190:0/4293198944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:29.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:29.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:29.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:29 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:27.865927+0000 mgr.smithi154.sllivz (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.521+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/4056240351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 msgr2=0x7fbbec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.521+0000 7fbbf0d7e700 1 --2- 172.21.15.190:0/4056240351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 0x7fbbec0f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fbbe0009fb0 tx=0x7fbbe0009b30).stop 2024-01-16T17:19:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.521+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/4056240351 shutdown_connections 2024-01-16T17:19:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.521+0000 7fbbf0d7e700 1 --2- 172.21.15.190:0/4056240351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 0x7fbbec0f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.521+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/4056240351 >> 172.21.15.190:0/4056240351 conn(0x7fbbec0f0b70 msgr2=0x7fbbec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.521+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/4056240351 shutdown_connections 2024-01-16T17:19:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.521+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/4056240351 wait complete. 2024-01-16T17:19:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.522+0000 7fbbf0d7e700 1 Processor -- start 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.522+0000 7fbbf0d7e700 1 -- start start 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.522+0000 7fbbf0d7e700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 0x7fbbec105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.522+0000 7fbbf0d7e700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec105b20 con 0x7fbbec0f5410 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.522+0000 7fbbea59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 0x7fbbec105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.522+0000 7fbbea59c700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 0x7fbbec105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:34132/0 (socket says 172.21.15.190:34132) 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.522+0000 7fbbea59c700 1 -- 172.21.15.190:0/2208928346 learned_addr learned my addr 172.21.15.190:0/2208928346 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.523+0000 7fbbea59c700 1 -- 172.21.15.190:0/2208928346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe0005040 con 0x7fbbec0f5410 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.523+0000 7fbbea59c700 1 --2- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 0x7fbbec105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fbbe0014000 tx=0x7fbbe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.523+0000 7fbbdaffd700 1 -- 172.21.15.190:0/2208928346 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbe00095f0 con 0x7fbbec0f5410 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.523+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec105d20 con 0x7fbbec0f5410 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.523+0000 7fbbdaffd700 1 -- 172.21.15.190:0/2208928346 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbe0007e70 con 0x7fbbec0f5410 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.523+0000 7fbbdaffd700 1 -- 172.21.15.190:0/2208928346 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbe0014ea0 con 0x7fbbec0f5410 2024-01-16T17:19:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.523+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbec1061b0 con 0x7fbbec0f5410 2024-01-16T17:19:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.524+0000 7fbbdaffd700 1 -- 172.21.15.190:0/2208928346 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbe0025070 con 0x7fbbec0f5410 2024-01-16T17:19:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.524+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbec045ad0 con 0x7fbbec0f5410 2024-01-16T17:19:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.524+0000 7fbbdaffd700 1 --2- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbbd4030000 0x7fbbd40324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.524+0000 7fbbdaffd700 1 -- 172.21.15.190:0/2208928346 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbe0040a00 con 0x7fbbec0f5410 2024-01-16T17:19:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.524+0000 7fbbe9d9b700 1 --2- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbbd4030000 0x7fbbd40324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.525+0000 7fbbe9d9b700 1 --2- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbbd4030000 0x7fbbd40324c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fbbdc0096a0 tx=0x7fbbdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:30.506 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.529+0000 7fbbdaffd700 1 -- 172.21.15.190:0/2208928346 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbe0026350 con 0x7fbbec0f5410 2024-01-16T17:19:30.713 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.736+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbec0fae60 con 0x7fbbec0f5410 2024-01-16T17:19:30.714 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.736+0000 7fbbdaffd700 1 -- 172.21.15.190:0/2208928346 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbbe0025300 con 0x7fbbec0f5410 2024-01-16T17:19:30.714 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:30.715 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbbd4030000 msgr2=0x7fbbd40324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 --2- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbbd4030000 0x7fbbd40324c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fbbdc0096a0 tx=0x7fbbdc006b40).stop 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 msgr2=0x7fbbec105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 --2- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 0x7fbbec105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fbbe0014000 tx=0x7fbbe000b040).stop 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 shutdown_connections 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 --2- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fbbd4030000 0x7fbbd40324c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 --2- 172.21.15.190:0/2208928346 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbec0f5410 0x7fbbec105630 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 >> 172.21.15.190:0/2208928346 conn(0x7fbbec0f0b70 msgr2=0x7fbbec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 shutdown_connections 2024-01-16T17:19:30.716 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:30.738+0000 7fbbf0d7e700 1 -- 172.21.15.190:0/2208928346 wait complete. 2024-01-16T17:19:30.717 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:31.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:29.866167+0000 mgr.smithi154.sllivz (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:31.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:31 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:30.737623+0000 mon.smithi154 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.190:0/2208928346' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:32.050 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:32.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.221+0000 7f8153fbf700 1 -- 172.21.15.190:0/3551978986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 msgr2=0x7f814c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.221+0000 7f8153fbf700 1 --2- 172.21.15.190:0/3551978986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 0x7f814c0f90f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f813c009fb0 tx=0x7f813c009b30).stop 2024-01-16T17:19:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.221+0000 7f8153fbf700 1 -- 172.21.15.190:0/3551978986 shutdown_connections 2024-01-16T17:19:33.202 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.221+0000 7f8153fbf700 1 --2- 172.21.15.190:0/3551978986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 0x7f814c0f90f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.221+0000 7f8153fbf700 1 -- 172.21.15.190:0/3551978986 >> 172.21.15.190:0/3551978986 conn(0x7f814c0f0b70 msgr2=0x7f814c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.221+0000 7f8153fbf700 1 -- 172.21.15.190:0/3551978986 shutdown_connections 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.221+0000 7f8153fbf700 1 -- 172.21.15.190:0/3551978986 wait complete. 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.222+0000 7f8153fbf700 1 Processor -- start 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.222+0000 7f8153fbf700 1 -- start start 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.222+0000 7f8153fbf700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 0x7f814c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.222+0000 7f8153fbf700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f814c106bd0 con 0x7f814c060280 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.222+0000 7f8151d5b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 0x7f814c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.222+0000 7f8151d5b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 0x7f814c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58394/0 (socket says 172.21.15.190:58394) 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.222+0000 7f8151d5b700 1 -- 172.21.15.190:0/3892180083 learned_addr learned my addr 172.21.15.190:0/3892180083 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.223+0000 7f8151d5b700 1 -- 172.21.15.190:0/3892180083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c005040 con 0x7f814c060280 2024-01-16T17:19:33.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.223+0000 7f8151d5b700 1 --2- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 0x7f814c1066e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f813c013510 tx=0x7f813c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.223+0000 7f81427fc700 1 -- 172.21.15.190:0/3892180083 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f813c009320 con 0x7f814c060280 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.223+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f814c106dd0 con 0x7f814c060280 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.223+0000 7f81427fc700 1 -- 172.21.15.190:0/3892180083 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f813c013f00 con 0x7f814c060280 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.223+0000 7f81427fc700 1 -- 172.21.15.190:0/3892180083 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f813c014dc0 con 0x7f814c060280 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.223+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f814c107260 con 0x7f814c060280 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.224+0000 7f81427fc700 1 -- 172.21.15.190:0/3892180083 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f813c014420 con 0x7f814c060280 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.224+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f814c045ad0 con 0x7f814c060280 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.224+0000 7f81427fc700 1 --2- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f813802ffb0 0x7f8138032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.224+0000 7f81427fc700 1 -- 172.21.15.190:0/3892180083 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f813c041110 con 0x7f814c060280 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.224+0000 7f815155a700 1 --2- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f813802ffb0 0x7f8138032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:33.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.225+0000 7f815155a700 1 --2- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f813802ffb0 0x7f8138032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f81480096a0 tx=0x7f8148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:33.206 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.229+0000 7f81427fc700 1 -- 172.21.15.190:0/3892180083 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f813c0095f0 con 0x7f814c060280 2024-01-16T17:19:33.411 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.434+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f814c108110 con 0x7f814c060280 2024-01-16T17:19:33.412 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.434+0000 7f81427fc700 1 -- 172.21.15.190:0/3892180083 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f813c01cb10 con 0x7f814c060280 2024-01-16T17:19:33.412 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:33.412 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:33.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f813802ffb0 msgr2=0x7f8138032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:33.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 --2- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f813802ffb0 0x7f8138032470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f81480096a0 tx=0x7f8148006b40).stop 2024-01-16T17:19:33.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 msgr2=0x7f814c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:33.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 --2- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 0x7f814c1066e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f813c013510 tx=0x7f813c00b040).stop 2024-01-16T17:19:33.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 shutdown_connections 2024-01-16T17:19:33.414 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 --2- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f813802ffb0 0x7f8138032470 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 --2- 172.21.15.190:0/3892180083 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f814c060280 0x7f814c1066e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 >> 172.21.15.190:0/3892180083 conn(0x7f814c0f0b70 msgr2=0x7f814c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 shutdown_connections 2024-01-16T17:19:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:33.436+0000 7f8153fbf700 1 -- 172.21.15.190:0/3892180083 wait complete. 2024-01-16T17:19:33.415 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:33.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:31.866425+0000 mgr.smithi154.sllivz (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:33.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:33 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:33.435518+0000 mon.smithi154 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.190:0/3892180083' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:34.786 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:34.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 -- 172.21.15.190:0/4060551019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 msgr2=0x7fba980f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 --2- 172.21.15.190:0/4060551019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 0x7fba980f90f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7fba80009fb0 tx=0x7fba80009b30).stop 2024-01-16T17:19:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 -- 172.21.15.190:0/4060551019 shutdown_connections 2024-01-16T17:19:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 --2- 172.21.15.190:0/4060551019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 0x7fba980f90f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 -- 172.21.15.190:0/4060551019 >> 172.21.15.190:0/4060551019 conn(0x7fba980f0b70 msgr2=0x7fba980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 -- 172.21.15.190:0/4060551019 shutdown_connections 2024-01-16T17:19:35.892 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 -- 172.21.15.190:0/4060551019 wait complete. 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 Processor -- start 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.913+0000 7fba9d07d700 1 -- start start 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.914+0000 7fba9d07d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 0x7fba981066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.914+0000 7fba9d07d700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba98106bd0 con 0x7fba98060280 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.914+0000 7fba96d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 0x7fba981066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.914+0000 7fba96d9d700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 0x7fba981066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58398/0 (socket says 172.21.15.190:58398) 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.914+0000 7fba96d9d700 1 -- 172.21.15.190:0/2101192918 learned_addr learned my addr 172.21.15.190:0/2101192918 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.914+0000 7fba96d9d700 1 -- 172.21.15.190:0/2101192918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80005040 con 0x7fba98060280 2024-01-16T17:19:35.893 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.914+0000 7fba96d9d700 1 --2- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 0x7fba981066e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fba80013510 tx=0x7fba8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:35.894 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.915+0000 7fba8f7fe700 1 -- 172.21.15.190:0/2101192918 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba80009320 con 0x7fba98060280 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.915+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba98106dd0 con 0x7fba98060280 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.915+0000 7fba8f7fe700 1 -- 172.21.15.190:0/2101192918 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba80013f00 con 0x7fba98060280 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.915+0000 7fba8f7fe700 1 -- 172.21.15.190:0/2101192918 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba80014dc0 con 0x7fba98060280 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.915+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba98107260 con 0x7fba98060280 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.915+0000 7fba8f7fe700 1 -- 172.21.15.190:0/2101192918 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba800095f0 con 0x7fba98060280 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.915+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba98045ad0 con 0x7fba98060280 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.916+0000 7fba8f7fe700 1 --2- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba8402ff60 0x7fba84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.916+0000 7fba8f7fe700 1 -- 172.21.15.190:0/2101192918 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba800410e0 con 0x7fba98060280 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.916+0000 7fba9659c700 1 --2- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba8402ff60 0x7fba84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:35.895 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.916+0000 7fba9659c700 1 --2- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba8402ff60 0x7fba84032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fba880096a0 tx=0x7fba88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:35.898 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:35.920+0000 7fba8f7fe700 1 -- 172.21.15.190:0/2101192918 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba80014420 con 0x7fba98060280 2024-01-16T17:19:35.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:35 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:33.866760+0000 mgr.smithi154.sllivz (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.128+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba981080b0 con 0x7fba98060280 2024-01-16T17:19:36.106 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.129+0000 7fba8f7fe700 1 -- 172.21.15.190:0/2101192918 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba8001c920 con 0x7fba98060280 2024-01-16T17:19:36.107 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:36.107 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:36.108 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba8402ff60 msgr2=0x7fba84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 --2- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba8402ff60 0x7fba84032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fba880096a0 tx=0x7fba88006b40).stop 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 msgr2=0x7fba981066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 --2- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 0x7fba981066e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fba80013510 tx=0x7fba8000b040).stop 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 shutdown_connections 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 --2- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fba8402ff60 0x7fba84032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 --2- 172.21.15.190:0/2101192918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba98060280 0x7fba981066e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 >> 172.21.15.190:0/2101192918 conn(0x7fba980f0b70 msgr2=0x7fba980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 shutdown_connections 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:36.131+0000 7fba9d07d700 1 -- 172.21.15.190:0/2101192918 wait complete. 2024-01-16T17:19:36.109 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:36.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:36 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:36.129975+0000 mon.smithi154 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.190:0/2101192918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:37.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:37.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:37.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:37 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:35.867096+0000 mgr.smithi154.sllivz (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:38.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.656+0000 7f8ed549a700 1 -- 172.21.15.190:0/984684510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 msgr2=0x7f8ed00f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:38.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.656+0000 7f8ed549a700 1 --2- 172.21.15.190:0/984684510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 0x7f8ed00f5df0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f8ec4009fb0 tx=0x7f8ec4009b30).stop 2024-01-16T17:19:38.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 -- 172.21.15.190:0/984684510 shutdown_connections 2024-01-16T17:19:38.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 --2- 172.21.15.190:0/984684510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 0x7f8ed00f5df0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:38.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 -- 172.21.15.190:0/984684510 >> 172.21.15.190:0/984684510 conn(0x7f8ed00f02e0 msgr2=0x7f8ed00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:38.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 -- 172.21.15.190:0/984684510 shutdown_connections 2024-01-16T17:19:38.635 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 -- 172.21.15.190:0/984684510 wait complete. 2024-01-16T17:19:38.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 Processor -- start 2024-01-16T17:19:38.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 -- start start 2024-01-16T17:19:38.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 0x7f8ed01055c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:38.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.657+0000 7f8ed549a700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0105ab0 con 0x7f8ed00f59d0 2024-01-16T17:19:38.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8eceffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 0x7f8ed01055c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:38.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8eceffd700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 0x7f8ed01055c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58422/0 (socket says 172.21.15.190:58422) 2024-01-16T17:19:38.636 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8eceffd700 1 -- 172.21.15.190:0/4043615494 learned_addr learned my addr 172.21.15.190:0/4043615494 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:38.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8eceffd700 1 -- 172.21.15.190:0/4043615494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec4005040 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8eceffd700 1 --2- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 0x7f8ed01055c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f8ec4013510 tx=0x7f8ec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8eb77fe700 1 -- 172.21.15.190:0/4043615494 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ec4009320 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8eb77fe700 1 -- 172.21.15.190:0/4043615494 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ec4013f00 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed0105cb0 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8eb77fe700 1 -- 172.21.15.190:0/4043615494 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ec4014d90 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.658+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed0106140 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.659+0000 7f8eb77fe700 1 -- 172.21.15.190:0/4043615494 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ec40095f0 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.659+0000 7f8eb77fe700 1 --2- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8eb802ff60 0x7f8eb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.659+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0045ad0 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.659+0000 7f8eb77fe700 1 -- 172.21.15.190:0/4043615494 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ec4040fc0 con 0x7f8ed00f59d0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.659+0000 7f8ece7fc700 1 --2- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8eb802ff60 0x7f8eb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.660+0000 7f8ece7fc700 1 --2- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8eb802ff60 0x7f8eb8032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8ec00096a0 tx=0x7f8ec0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:38.641 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.664+0000 7f8eb77fe700 1 -- 172.21.15.190:0/4043615494 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ec4014420 con 0x7f8ed00f59d0 2024-01-16T17:19:38.849 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.871+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ed0106f70 con 0x7f8ed00f59d0 2024-01-16T17:19:38.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.872+0000 7f8eb77fe700 1 -- 172.21.15.190:0/4043615494 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ec4010230 con 0x7f8ed00f59d0 2024-01-16T17:19:38.850 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:38.850 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:38.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8eb802ff60 msgr2=0x7f8eb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:38.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 --2- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8eb802ff60 0x7f8eb8032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8ec00096a0 tx=0x7f8ec0006b40).stop 2024-01-16T17:19:38.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 msgr2=0x7f8ed01055c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:38.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 --2- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 0x7f8ed01055c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f8ec4013510 tx=0x7f8ec400b040).stop 2024-01-16T17:19:38.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 shutdown_connections 2024-01-16T17:19:38.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 --2- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7f8eb802ff60 0x7f8eb8032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:38.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 --2- 172.21.15.190:0/4043615494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ed00f59d0 0x7f8ed01055c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:38.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 >> 172.21.15.190:0/4043615494 conn(0x7f8ed00f02e0 msgr2=0x7f8ed00f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:38.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 shutdown_connections 2024-01-16T17:19:38.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:38.874+0000 7f8ed549a700 1 -- 172.21.15.190:0/4043615494 wait complete. 2024-01-16T17:19:38.852 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:39.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:37.867427+0000 mgr.smithi154.sllivz (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:39.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:39 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:38.873198+0000 mon.smithi154 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.190:0/4043615494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:40.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:40.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:41.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.457+0000 7fa26ae4b700 1 -- 172.21.15.190:0/3019052051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 msgr2=0x7fa2640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:41.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.457+0000 7fa26ae4b700 1 --2- 172.21.15.190:0/3019052051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 0x7fa2640f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fa258009fb0 tx=0x7fa258009b30).stop 2024-01-16T17:19:41.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.457+0000 7fa26ae4b700 1 -- 172.21.15.190:0/3019052051 shutdown_connections 2024-01-16T17:19:41.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.457+0000 7fa26ae4b700 1 --2- 172.21.15.190:0/3019052051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 0x7fa2640f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:41.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.457+0000 7fa26ae4b700 1 -- 172.21.15.190:0/3019052051 >> 172.21.15.190:0/3019052051 conn(0x7fa2640f0b70 msgr2=0x7fa2640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:41.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.457+0000 7fa26ae4b700 1 -- 172.21.15.190:0/3019052051 shutdown_connections 2024-01-16T17:19:41.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.457+0000 7fa26ae4b700 1 -- 172.21.15.190:0/3019052051 wait complete. 2024-01-16T17:19:41.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.458+0000 7fa26ae4b700 1 Processor -- start 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.458+0000 7fa26ae4b700 1 -- start start 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.458+0000 7fa26ae4b700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 0x7fa264105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.458+0000 7fa26ae4b700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa264105b20 con 0x7fa2640f5410 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.458+0000 7fa268be7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 0x7fa264105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.458+0000 7fa268be7700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 0x7fa264105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:58438/0 (socket says 172.21.15.190:58438) 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.458+0000 7fa268be7700 1 -- 172.21.15.190:0/464955353 learned_addr learned my addr 172.21.15.190:0/464955353 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.459+0000 7fa268be7700 1 -- 172.21.15.190:0/464955353 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa258005040 con 0x7fa2640f5410 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.459+0000 7fa268be7700 1 --2- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 0x7fa264105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fa258013510 tx=0x7fa25800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.459+0000 7fa2617fa700 1 -- 172.21.15.190:0/464955353 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa258009320 con 0x7fa2640f5410 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.459+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa264105d20 con 0x7fa2640f5410 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.459+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2641061b0 con 0x7fa2640f5410 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.459+0000 7fa2617fa700 1 -- 172.21.15.190:0/464955353 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa258013f00 con 0x7fa2640f5410 2024-01-16T17:19:41.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.459+0000 7fa2617fa700 1 -- 172.21.15.190:0/464955353 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa258014d90 con 0x7fa2640f5410 2024-01-16T17:19:41.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.460+0000 7fa2617fa700 1 -- 172.21.15.190:0/464955353 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa258014420 con 0x7fa2640f5410 2024-01-16T17:19:41.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.460+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa264045ad0 con 0x7fa2640f5410 2024-01-16T17:19:41.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.460+0000 7fa2617fa700 1 --2- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa2540343d0 0x7fa254036890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:41.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.460+0000 7fa2617fa700 1 -- 172.21.15.190:0/464955353 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa258040fd0 con 0x7fa2640f5410 2024-01-16T17:19:41.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.460+0000 7fa263fff700 1 --2- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa2540343d0 0x7fa254036890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:41.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.461+0000 7fa263fff700 1 --2- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa2540343d0 0x7fa254036890 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa2500096a0 tx=0x7fa250006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:41.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.464+0000 7fa2617fa700 1 -- 172.21.15.190:0/464955353 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa25806d2b0 con 0x7fa2640f5410 2024-01-16T17:19:41.652 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.674+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2640fae60 con 0x7fa2640f5410 2024-01-16T17:19:41.653 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.674+0000 7fa2617fa700 1 -- 172.21.15.190:0/464955353 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa2580095f0 con 0x7fa2640f5410 2024-01-16T17:19:41.653 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:41.653 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:41.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa2540343d0 msgr2=0x7fa254036890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:41.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 --2- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa2540343d0 0x7fa254036890 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa2500096a0 tx=0x7fa250006b40).stop 2024-01-16T17:19:41.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 msgr2=0x7fa264105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:41.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 --2- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 0x7fa264105630 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fa258013510 tx=0x7fa25800b040).stop 2024-01-16T17:19:41.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 shutdown_connections 2024-01-16T17:19:41.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 --2- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fa2540343d0 0x7fa254036890 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:41.655 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 --2- 172.21.15.190:0/464955353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa2640f5410 0x7fa264105630 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:41.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 >> 172.21.15.190:0/464955353 conn(0x7fa2640f0b70 msgr2=0x7fa2640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:41.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 shutdown_connections 2024-01-16T17:19:41.656 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:41.676+0000 7fa26ae4b700 1 -- 172.21.15.190:0/464955353 wait complete. 2024-01-16T17:19:41.656 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:41.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:39.867746+0000 mgr.smithi154.sllivz (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:41.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:41 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:41.676620+0000 mon.smithi154 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.190:0/464955353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:43.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:43.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:43.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:43 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:41.868144+0000 mgr.smithi154.sllivz (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:44.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.193+0000 7fb4c4436700 1 -- 172.21.15.190:0/2733625225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 msgr2=0x7fb4bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:44.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.193+0000 7fb4c4436700 1 --2- 172.21.15.190:0/2733625225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 0x7fb4bc0f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fb4ac009fb0 tx=0x7fb4ac009b30).stop 2024-01-16T17:19:44.172 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 -- 172.21.15.190:0/2733625225 shutdown_connections 2024-01-16T17:19:44.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 --2- 172.21.15.190:0/2733625225 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 0x7fb4bc0f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:44.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 -- 172.21.15.190:0/2733625225 >> 172.21.15.190:0/2733625225 conn(0x7fb4bc0f0b70 msgr2=0x7fb4bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:44.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 -- 172.21.15.190:0/2733625225 shutdown_connections 2024-01-16T17:19:44.173 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 -- 172.21.15.190:0/2733625225 wait complete. 2024-01-16T17:19:44.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 Processor -- start 2024-01-16T17:19:44.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 -- start start 2024-01-16T17:19:44.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 0x7fb4bc109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:44.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.194+0000 7fb4c4436700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4bc060240 con 0x7fb4bc0f5410 2024-01-16T17:19:44.174 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.195+0000 7fb4c21d2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 0x7fb4bc109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.195+0000 7fb4c21d2700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 0x7fb4bc109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60260/0 (socket says 172.21.15.190:60260) 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.195+0000 7fb4c21d2700 1 -- 172.21.15.190:0/3451774904 learned_addr learned my addr 172.21.15.190:0/3451774904 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.195+0000 7fb4c21d2700 1 -- 172.21.15.190:0/3451774904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ac005040 con 0x7fb4bc0f5410 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.195+0000 7fb4c21d2700 1 --2- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 0x7fb4bc109400 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb4ac013510 tx=0x7fb4ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.195+0000 7fb4b2ffd700 1 -- 172.21.15.190:0/3451774904 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4ac009320 con 0x7fb4bc0f5410 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.195+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4bc060440 con 0x7fb4bc0f5410 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.195+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4bc109ab0 con 0x7fb4bc0f5410 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.196+0000 7fb4b2ffd700 1 -- 172.21.15.190:0/3451774904 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4ac013f00 con 0x7fb4bc0f5410 2024-01-16T17:19:44.175 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.196+0000 7fb4b2ffd700 1 -- 172.21.15.190:0/3451774904 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4ac014ce0 con 0x7fb4bc0f5410 2024-01-16T17:19:44.176 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.196+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a4004fa0 con 0x7fb4bc0f5410 2024-01-16T17:19:44.176 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.196+0000 7fb4b2ffd700 1 -- 172.21.15.190:0/3451774904 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4ac014420 con 0x7fb4bc0f5410 2024-01-16T17:19:44.176 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.196+0000 7fb4b2ffd700 1 --2- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb4a802ff60 0x7fb4a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:44.176 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.197+0000 7fb4b2ffd700 1 -- 172.21.15.190:0/3451774904 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4ac040ed0 con 0x7fb4bc0f5410 2024-01-16T17:19:44.176 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.197+0000 7fb4c19d1700 1 --2- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb4a802ff60 0x7fb4a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:44.176 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.197+0000 7fb4c19d1700 1 --2- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb4a802ff60 0x7fb4a8032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb4b80096a0 tx=0x7fb4b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:44.179 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.201+0000 7fb4b2ffd700 1 -- 172.21.15.190:0/3451774904 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4ac014e40 con 0x7fb4bc0f5410 2024-01-16T17:19:44.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.409+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4a4005670 con 0x7fb4bc0f5410 2024-01-16T17:19:44.389 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.410+0000 7fb4b2ffd700 1 -- 172.21.15.190:0/3451774904 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb4ac012750 con 0x7fb4bc0f5410 2024-01-16T17:19:44.389 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:44.389 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:44.390 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.411+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb4a802ff60 msgr2=0x7fb4a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:44.390 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.411+0000 7fb4c4436700 1 --2- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb4a802ff60 0x7fb4a8032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb4b80096a0 tx=0x7fb4b8006b40).stop 2024-01-16T17:19:44.390 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.411+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 msgr2=0x7fb4bc109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:44.390 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.411+0000 7fb4c4436700 1 --2- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 0x7fb4bc109400 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fb4ac013510 tx=0x7fb4ac00b040).stop 2024-01-16T17:19:44.390 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.411+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 shutdown_connections 2024-01-16T17:19:44.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.411+0000 7fb4c4436700 1 --2- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fb4a802ff60 0x7fb4a8032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:44.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.411+0000 7fb4c4436700 1 --2- 172.21.15.190:0/3451774904 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4bc0f5410 0x7fb4bc109400 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:44.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.411+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 >> 172.21.15.190:0/3451774904 conn(0x7fb4bc0f0b70 msgr2=0x7fb4bc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:44.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.412+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 shutdown_connections 2024-01-16T17:19:44.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:44.412+0000 7fb4c4436700 1 -- 172.21.15.190:0/3451774904 wait complete. 2024-01-16T17:19:44.391 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:44.955 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:44 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: audit 2024-01-16T17:19:44.412133+0000 mon.smithi154 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.190:0/3451774904' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-16T17:19:45.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-16T17:19:45.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 059f8940-b492-11ee-95ae-87774f69a715 -- ceph mon dump -f json 2024-01-16T17:19:45.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:45 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: cluster 2024-01-16T17:19:43.868500+0000 mgr.smithi154.sllivz (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-16T17:19:46.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.871+0000 7fd6e71c9700 1 -- 172.21.15.190:0/866660953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 msgr2=0x7fd6e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:46.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.871+0000 7fd6e71c9700 1 --2- 172.21.15.190:0/866660953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 0x7fd6e00f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fd6d4009fb0 tx=0x7fd6d4009b30).stop 2024-01-16T17:19:46.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.871+0000 7fd6e71c9700 1 -- 172.21.15.190:0/866660953 shutdown_connections 2024-01-16T17:19:46.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.871+0000 7fd6e71c9700 1 --2- 172.21.15.190:0/866660953 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 0x7fd6e00f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:46.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.871+0000 7fd6e71c9700 1 -- 172.21.15.190:0/866660953 >> 172.21.15.190:0/866660953 conn(0x7fd6e00f0b70 msgr2=0x7fd6e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:46.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.871+0000 7fd6e71c9700 1 -- 172.21.15.190:0/866660953 shutdown_connections 2024-01-16T17:19:46.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.871+0000 7fd6e71c9700 1 -- 172.21.15.190:0/866660953 wait complete. 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.872+0000 7fd6e71c9700 1 Processor -- start 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.872+0000 7fd6e71c9700 1 -- start start 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.872+0000 7fd6e71c9700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 0x7fd6e0106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.872+0000 7fd6e71c9700 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e0106c00 con 0x7fd6e00f5410 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.872+0000 7fd6e4f65700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 0x7fd6e0106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.872+0000 7fd6e4f65700 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 0x7fd6e0106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.190:60280/0 (socket says 172.21.15.190:60280) 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.872+0000 7fd6e4f65700 1 -- 172.21.15.190:0/2344596057 learned_addr learned my addr 172.21.15.190:0/2344596057 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.873+0000 7fd6e4f65700 1 -- 172.21.15.190:0/2344596057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d4005040 con 0x7fd6e00f5410 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.873+0000 7fd6e4f65700 1 --2- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 0x7fd6e0106710 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fd6d4014000 tx=0x7fd6d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.873+0000 7fd6dd7fa700 1 -- 172.21.15.190:0/2344596057 <== mon.0 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6d40095f0 con 0x7fd6e00f5410 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.873+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e0106e00 con 0x7fd6e00f5410 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.873+0000 7fd6dd7fa700 1 -- 172.21.15.190:0/2344596057 <== mon.0 v2:172.21.15.154:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6d4007e70 con 0x7fd6e00f5410 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.873+0000 7fd6dd7fa700 1 -- 172.21.15.190:0/2344596057 <== mon.0 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6d4014ea0 con 0x7fd6e00f5410 2024-01-16T17:19:46.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.873+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e0107290 con 0x7fd6e00f5410 2024-01-16T17:19:46.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.874+0000 7fd6dd7fa700 1 -- 172.21.15.190:0/2344596057 <== mon.0 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6d4025070 con 0x7fd6e00f5410 2024-01-16T17:19:46.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.874+0000 7fd6dd7fa700 1 --2- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd6d002ff60 0x7fd6d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-16T17:19:46.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.874+0000 7fd6dd7fa700 1 -- 172.21.15.190:0/2344596057 <== mon.0 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd6d4040a40 con 0x7fd6e00f5410 2024-01-16T17:19:46.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.874+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e0045ad0 con 0x7fd6e00f5410 2024-01-16T17:19:46.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.874+0000 7fd6dffff700 1 --2- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd6d002ff60 0x7fd6d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-16T17:19:46.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.875+0000 7fd6dffff700 1 --2- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd6d002ff60 0x7fd6d0032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd6cc0096a0 tx=0x7fd6cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T17:19:46.857 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:46.879+0000 7fd6dd7fa700 1 -- 172.21.15.190:0/2344596057 <== mon.0 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6d4026350 con 0x7fd6e00f5410 2024-01-16T17:19:47.065 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.087+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6e00faf50 con 0x7fd6e00f5410 2024-01-16T17:19:47.066 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.087+0000 7fd6dd7fa700 1 -- 172.21.15.190:0/2344596057 <== mon.0 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd6d4012330 con 0x7fd6e00f5410 2024-01-16T17:19:47.067 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-01-16T17:19:47.067 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"059f8940-b492-11ee-95ae-87774f69a715","modified":"2024-01-16T17:10:03.667056Z","created":"2024-01-16T17:10:03.667056Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi154","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-16T17:19:47.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd6d002ff60 msgr2=0x7fd6d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:47.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 --2- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd6d002ff60 0x7fd6d0032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd6cc0096a0 tx=0x7fd6cc006b40).stop 2024-01-16T17:19:47.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 msgr2=0x7fd6e0106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T17:19:47.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 --2- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 0x7fd6e0106710 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fd6d4014000 tx=0x7fd6d400b040).stop 2024-01-16T17:19:47.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 shutdown_connections 2024-01-16T17:19:47.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 --2- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:6800/2708438664,v1:172.21.15.154:6801/2708438664] conn(0x7fd6d002ff60 0x7fd6d0032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:47.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 --2- 172.21.15.190:0/2344596057 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6e00f5410 0x7fd6e0106710 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-16T17:19:47.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 >> 172.21.15.190:0/2344596057 conn(0x7fd6e00f0b70 msgr2=0x7fd6e005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T17:19:47.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 shutdown_connections 2024-01-16T17:19:47.069 INFO:teuthology.orchestra.run.smithi190.stderr:2024-01-16T17:19:47.089+0000 7fd6e71c9700 1 -- 172.21.15.190:0/2344596057 wait complete. 2024-01-16T17:19:47.069 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-01-16T17:19:47.446 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-16T17:19:47.448 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-16T17:19:47.448 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-16T17:19:47.472 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-16T17:19:47.491 INFO:tasks.cephadm:Stopping all daemons... 2024-01-16T17:19:47.491 INFO:tasks.cephadm.mon.smithi154:Stopping mon.smithi154... 2024-01-16T17:19:47.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154 2024-01-16T17:19:47.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:47 smithi154 systemd[1]: Stopping Ceph mon.smithi154 for 059f8940-b492-11ee-95ae-87774f69a715... 2024-01-16T17:19:47.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:19:47.641+0000 7faacc9cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi154 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-16T17:19:47.956 INFO:journalctl@ceph.mon.smithi154.smithi154.stdout:Jan 16 17:19:47 smithi154 ceph-059f8940-b492-11ee-95ae-87774f69a715-mon.smithi154[67522]: debug 2024-01-16T17:19:47.641+0000 7faacc9cc700 -1 mon.smithi154@0(leader) e1 *** Got Signal Terminated *** 2024-01-16T17:19:48.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi154.service' 2024-01-16T17:19:48.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-16T17:19:48.172 INFO:tasks.cephadm.mon.smithi154:Stopped mon.smithi154 2024-01-16T17:19:48.172 INFO:tasks.cephadm.mon.smithi190:Stopping mon.smithi190... 2024-01-16T17:19:48.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi190 2024-01-16T17:19:48.207 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi190.service: Unit ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi190.service not loaded. 2024-01-16T17:19:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-16T17:19:48.209 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi190" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 5: 'sudo systemctl stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi190' 2024-01-16T17:19:48.210 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-16T17:19:48.212 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255/remote/ubuntu@smithi154.front.sepia.ceph.com/crash 2024-01-16T17:19:48.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/crash -- . 2024-01-16T17:19:48.265 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255/remote/ubuntu@smithi190.front.sepia.ceph.com/crash 2024-01-16T17:19:48.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/crash -- . 2024-01-16T17:19:48.302 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/059f8940-b492-11ee-95ae-87774f69a715/crash: Cannot open: No such file or directory 2024-01-16T17:19:48.302 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-01-16T17:19:48.303 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-16T17:19:48.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-16T17:19:48.333 INFO:teuthology.orchestra.run.smithi154.stderr:grep: /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph.log: No such file or directory 2024-01-16T17:19:48.335 INFO:tasks.cephadm:Compressing logs... 2024-01-16T17:19:48.335 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T17:19:48.377 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T17:19:48.401 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-16T17:19:48.401 INFO:teuthology.orchestra.run.smithi154.stderr:: No such file or directory 2024-01-16T17:19:48.402 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-mon.smithi154.log 2024-01-16T17:19:48.402 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-volume.log 2024-01-16T17:19:48.402 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-mgr.smithi154.sllivz.log 2024-01-16T17:19:48.403 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-mon.smithi154.log: /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-volume.log: /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-mgr.smithi154.sllivz.log: 88.4% -- replaced with /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-volume.log.gz 2024-01-16T17:19:48.404 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-16T17:19:48.404 INFO:teuthology.orchestra.run.smithi190.stderr:: No such file or directory 2024-01-16T17:19:48.406 INFO:teuthology.orchestra.run.smithi154.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-16T17:19:48.407 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cephadm.log: 96.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-16T17:19:48.408 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-01-16T17:19:48.408 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m0.014s 2024-01-16T17:19:48.408 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m0.014s 2024-01-16T17:19:48.408 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m0.009s 2024-01-16T17:19:48.485 INFO:teuthology.orchestra.run.smithi154.stderr: 92.7% -- replaced with /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-mgr.smithi154.sllivz.log.gz 2024-01-16T17:19:48.550 INFO:teuthology.orchestra.run.smithi154.stderr: 90.5% -- replaced with /var/log/ceph/059f8940-b492-11ee-95ae-87774f69a715/ceph-mon.smithi154.log.gz 2024-01-16T17:19:48.552 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-01-16T17:19:48.552 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m0.161s 2024-01-16T17:19:48.552 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m0.242s 2024-01-16T17:19:48.552 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.015s 2024-01-16T17:19:48.552 INFO:tasks.cephadm:Archiving logs... 2024-01-16T17:19:48.553 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255/remote/ubuntu@smithi154.front.sepia.ceph.com/log 2024-01-16T17:19:48.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T17:19:48.666 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255/remote/ubuntu@smithi190.front.sepia.ceph.com/log 2024-01-16T17:19:48.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T17:19:48.702 INFO:tasks.cephadm:Removing cluster... 2024-01-16T17:19:48.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 059f8940-b492-11ee-95ae-87774f69a715 --force 2024-01-16T17:20:53.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 059f8940-b492-11ee-95ae-87774f69a715 --force 2024-01-16T17:20:53.682 INFO:tasks.cephadm:Removing cephadm ... 2024-01-16T17:20:53.683 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-16T17:20:53.702 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-16T17:20:53.719 INFO:tasks.cephadm:Teardown complete 2024-01-16T17:20:53.719 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 5: 'sudo systemctl stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi190' 2024-01-16T17:20:53.927 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ab074bccc3304ea9a7e19184e550354b Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_a70aebef70df6ec2d957347ccbfc484e0573dd5f/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi190 with status 5: 'sudo systemctl stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi190' 2024-01-16T17:20:53.931 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-16T17:20:53.940 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T17:20:54.073 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T17:20:54.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T17:20:54.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T17:20:54.166 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T17:20:54.174 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T17:20:54.174 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T17:20:54.177 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T17:20:54.192 INFO:teuthology.orchestra.run.smithi154.stderr:bash: ntpq: command not found 2024-01-16T17:20:54.193 INFO:teuthology.orchestra.run.smithi190.stderr:bash: ntpq: command not found 2024-01-16T17:20:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T17:20:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-01-16T17:20:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +2847us[+2847us] +/- 109ms 2024-01-16T17:20:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -686us[ -643us] +/- 69ms 2024-01-16T17:20:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -1118us[-1118us] +/- 78ms 2024-01-16T17:20:54.197 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:20:54.198 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T17:20:54.199 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-01-16T17:20:54.199 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +2821us[+2821us] +/- 109ms 2024-01-16T17:20:54.199 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 65 -688us[ -630us] +/- 68ms 2024-01-16T17:20:54.199 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -1183us[-1183us] +/- 78ms 2024-01-16T17:20:54.199 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T17:20:54.199 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T17:20:54.208 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T17:20:54.209 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T17:20:54.219 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T17:20:54.299 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T17:20:54.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T17:20:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T17:20:54.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T17:20:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T17:20:54.421 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T17:20:54.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T17:20:54.441 INFO:teuthology.task.internal:Duration was 1263.704862 seconds 2024-01-16T17:20:54.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T17:20:54.457 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T17:20:54.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T17:20:54.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T17:20:54.503 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T17:20:54.506 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T17:20:54.717 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-16T17:20:54.718 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-01-16T17:20:54.718 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T17:20:54.740 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-01-16T17:20:54.740 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T17:20:54.762 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-16T17:20:54.762 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T17:20:54.782 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T17:20:54.851 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-16T17:20:54.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T17:20:54.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T17:20:54.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T17:20:54.949 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T17:20:54.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T17:20:54.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T17:20:55.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T17:20:55.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T17:20:55.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T17:20:55.044 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-01-16T17:20:55.070 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-01-16T17:20:55.114 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T17:20:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T17:20:55.131 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T17:20:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T17:20:55.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T17:20:55.181 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T17:20:55.182 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255/remote/smithi154 2024-01-16T17:20:55.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T17:20:55.222 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255/remote/smithi190 2024-01-16T17:20:55.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T17:20:55.264 INFO:teuthology.task.internal:Removing archive directory... 2024-01-16T17:20:55.264 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T17:20:55.267 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T17:20:55.315 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T17:20:55.326 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T17:20:55.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T17:20:55.337 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T17:20:55.337 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T17:20:55.339 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T17:20:55.354 INFO:teuthology.orchestra.run.smithi154.stdout: 525352 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 17:20 /home/ubuntu/cephtest 2024-01-16T17:20:55.372 INFO:teuthology.orchestra.run.smithi190.stdout: 525352 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 17:20 /home/ubuntu/cephtest 2024-01-16T17:20:55.374 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T17:20:55.383 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T17:20:55.407 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T17:20:55.430 DEBUG:teuthology.nuke:shortname: smithi154 2024-01-16T17:20:55.431 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T17:20:55.452 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 15:58:32.976180', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCI5KUKyPq1BeVf1ZpBGEQk2yziu3e/9ybt2Y1uGQayLO2NHgYzXWz2umcQazanfvGyXCGDGmnehKnWACr4g9DQ='} 2024-01-16T17:20:55.453 DEBUG:teuthology.nuke:shortname: smithi190 2024-01-16T17:20:55.453 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T17:20:55.477 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-16_15:55:17-rados-wip-yuri5-testing-2024-01-11-1300-pacific-distro-default-smithi/7518255', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 15:58:32.977597', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKZhfNIRpxTyWntFUjOkAB/4xt0JjrYVa5smZzrD8Dl+c71lylpOUdl7IoEp33XwzaS5Cw/24e5iW0TRVSSuwLs='} 2024-01-16T17:20:55.504 INFO:teuthology.orchestra.console.smithi154:Power off 2024-01-16T17:20:55.504 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T17:20:55.526 DEBUG:teuthology.orchestra.console.smithi154:power off output: Chassis Power Control: Down/Off 2024-01-16T17:20:55.526 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T17:20:55.539 INFO:teuthology.orchestra.console.smithi190:Power off 2024-01-16T17:20:55.539 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T17:20:55.548 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-01-16T17:20:55.561 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2024-01-16T17:20:55.561 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T17:20:55.584 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-01-16T17:20:59.549 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T17:20:59.585 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T17:20:59.659 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is on 2024-01-16T17:20:59.694 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-01-16T17:21:03.665 DEBUG:teuthology.orchestra.console.smithi154:pexpect command: ipmitool -H smithi154.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T17:21:03.696 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T17:21:03.775 DEBUG:teuthology.orchestra.console.smithi154:check power output: Chassis Power is off 2024-01-16T17:21:03.804 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-01-16T17:21:03.876 INFO:teuthology.orchestra.console.smithi154:Power off completed 2024-01-16T17:21:07.808 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T17:21:07.918 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2024-01-16T17:21:08.019 INFO:teuthology.orchestra.console.smithi190:Power off completed 2024-01-16T17:21:08.123 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1263.704862356186 failure_reason: 'Command failed on smithi190 with status 5: ''sudo systemctl stop ceph-059f8940-b492-11ee-95ae-87774f69a715@mon.smithi190''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ab074bccc3304ea9a7e19184e550354b status: fail success: false 2024-01-16T17:21:08.123 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T17:21:08.209 INFO:teuthology.run:FAIL